As a brand-new auditor I remember receiving giant text or excel files full of data I couldn’t decipher. Inside the file would be hundreds (or more likely thousands) of line items. I was too naive and not confident enough to question what the client had provided – so I accepted it without asking a single question about how it was generated.
After working in the field for a few years and speaking with countless other auditors, I have come to realize this problem is rampant in the IT audit field – especially in the case of testing change management.
If you have ever been audited (or are an auditor yourself), you have probably produced or requested “a complete population of changes to XYZ application from date to date“.
The Auditor’s plan here is to choose a small sample of changes from the total population for auditing purposes. By choosing a small sample from a complete population, the auditor can statistically make reasonable assumptions about the population as a whole without burdening the auditee with requests for potentially hundreds or thousands of changes.
The key here is that the population has to be accurate and complete.
What happens if the population is corrupt? What happens if the population only includes changes that followed the change management process, but ignores those that skipped it (Like those that weren’t entered into the ticketing system!)? And how can an auditor ensure that the population is complete?
Verifying Population Completeness
Here are a few tools and questions that should help assure the auditor that the population of changes is complete.
1. Built-in or Custom Query: Understand if the query is built-in to the application (as in part of the application’s features) or custom-built by the developer. As an auditor we have greater confidence in built-in features than those developed by a individual as an add-on (it is typically more difficult to ensure quality and accuracy of the query since it hasn’t been as well vetted).
If the query was custom-built by an in-house developer you should spend a few minutes walking through the query structure (even if you don’t fully understand the complexities) to get a sense for the developer’s thought process. Often, problems with the query will quickly surface – especially if the only time the company uses the query is for the audit (which we see pretty often). It’s a red flag if the developer doesn’t seem confident or familiar with their own work.
2. Query Parameters: Whether the query is built-in or custom-built, you should always inspect the parameters. Ensure that the query includes all applications in scope as well as the date range in question.
1. From the Ticketing System: The first place most developers go to grab a population of changes is usually the change ticketing system. This makes sense because the ticketing system (hypothetically) has the most detailed information about a given change and is by far the most user-friendly source to procure information (especially when it comes to audit trails).
Unfortunately, as an auditor you DO NOT want to rely on the ticketing system for your population of changes.
Changes from the ticketing system are inherently flawed from an audit perspective because anything entered in the ticketing system should have (hypothetically) followed the change management process. (This is sometimes called testing the affirmative – or only testing those that are likely to pass.) What the ticketing system ignores are those changes that bypassed the change management process and are thus, more likely to have high risks and audit findings attached to them.
The ticketing system is a great place to start, but only the first step in gathering population details. The next two options should help validate what you are seeing in the ticketing system.
2. From the source code development tool: Most source code management tools have a method to generate all changes implemented to production (often called “change commits”). Tools like Subversion, Jenkins, Git, and various add-ons are worth researching as they are commonly utilized to track deployments. This is a highly effective way to choose changes because the list of implementations is comprehensive and not easily manipulated.
Challenges: If you use the Source Code Management Tool (SCMT) as a source of your population, be aware there are several challenges including:
- Tracing a change in the SCMT (which is often in an unorganized/non-descriptive form) back to the change management documentation in the ticketing system (which is usually more organized).
- Filtering the changes in the SCMT to the appropriate scope (filtering for the appropriate apps and dates can be difficult).
- If your scope includes multiple applications there could be more than one SCMT in which you have to familiarize yourself to effectively inspect.
3. Modification date on live executable files: Another great place you should check for changes is the modification dates attached to the application executable files. This information is typically available (i.e. in Microsoft Windows) in the folder where the files are housed in the “Date modified” column.
Challenges: There are certain limitations on utilizing the Executable files for change populations. For one, often the executable file typically only shows the last date modified, not all modifications. This pretty much limits the usefulness of this method to verifying when a clients says that there have been no changes during the review period, but ineffective for pulling a full population.
However, I recommend checking the last modification date as be an easy way to verify that the information you see from the SCMT and the ticketing system are in-line.
Hopefully these suggestions add a few tools to your auditing toolkit. Are we missing anything?