Audit Log Analyzer idea

Hello everyone! I am a part of UW Impact++ team, and we are trying to make something useful for the ODK community!

We noticed that there are a lot of insights that can be discovered from the audit log files associated with the submissions (such as how long each question takes and so on). And we are trying to build something that can take in the csv file that contains the audit log, and produce something that will be useful for you.

Exactly how this gets built depends on all of your voices and inputs!! For example, we still haven't decided on the exact format of the software we want to build. Since the audit file contains very sensitive information, we want to make a local app running on the desktop, rather than creating a web app.

There are a couple of options here:

  1. build a plain simple python script -- this assumes people are comfortable with command lines.
  2. build a JAR with some frontend such as buttons and interface for uploading the file
  3. Make a webservice that only displays a javascript frontend, and the data doesn’t go to the server and is processed locally. (Using something like Vue or React?)

Also, how complicated this software ends up being and what analytics it comes with also depends on our ODK community! Currently we have a python script that takes in the audit.csv file and outputs the average time of filling out each question by the data collectors.

Please let us know any ideas you have with respect to the scope and the format of this software. We are trying to make sure what we build would actually be useful to our community. Thanks a ton!

Great idea!

A number of scenarios from the audit log file could be very useful in understanding data collectors challenges, behavior or "tricks".

One useful output is being able to identify a trend on responses i.e time taken to administer a survey. It could be possible to tell if the data collector is entering data from recall or interviewing someone.

Researchers could also be able to potentially identify questions field staff could be having problems with by looking at time taken.

I hope the proposed app/tool will provide graphical output to monitor trends.

My thoughts...

Paul

2 Likes

Hello Paul! Thank you so much for your thoughts!! I don't know if I completely understand it, but here is my understanding of your ideas:

it would be cool to generate a plot of how long a survey takes to completion as y-axis and the survey number as the x-axis? That is, the plot will show that the first survey takes 1 minute to complete, second survey takes 2 minute, third survey take 1 minute, and so on.

This can also be extended to individual questions -- to see how long each question takes to be answered as more and more surveys are being collected.

Do I have the right idea here?

Best,
Hugh

1 Like

Yes!

You have the right idea

Paul

2 Likes

@hugh_sun

One other thing, the audit log (I hope) documents a data collector going back and forth changing responses. I wish the proposed tool can be able to also show this in a useful way.....

This would be useful in data QC and QA

Paul

2 Likes

Thank you for mentioning this! We are currently flagging the questions whose answer gets changed a lot, but in the future we want to support flagging the data collectors that change answers frequently.

Also, we are trying to have a prototype ready in a couple of weeks. When it is ready, would you like to our first test user? We greatly appreciate your involvement!

Hugh

Great!

Please share the prototype when ready.

Paul

@hugh_sun thanks for starting this, analyzing the audit log is something I definitely would like to explore, but I am lacking the time and resources to do so. In general it think this is very important to objectivise how forms are filled out. I fully agree with all great @paul_macharia's suggestions.

A few additions, elaborating around what has already been mentioned, split between indicators related to

  • individual questions

    • identify questions for which answers are regularly modified by data collectors
      From a data management perspective this may indicate that the phrasing of the question / answers is too ambiguous. This would lead me to consider the quality - accuracy- of the answers as not very high and be very cautious when interpreting findings related to such a question.
    • identify questions which take more time for data collectors to answer, as this could allow to improve forms / focus on where bottlenecks have been identified. When you have several questions around the same thematic and need to prioritise, this indicator may help to make operational decisions
    • identify if an answer was modified a "very long time" after the first entry (very long time depending on what is defined as acceptable by the context of the study), e.g. it could be any modification occurring after a few seconds if you are trying to collect timings, after 1h or on a different day in other data collection
    • identify if a question that should be answered at a specific location was not answered where it should (could also help identify if the survey is completed from recall vs. while interviewing a respondent or if an observation was not direct)
  • sequence of questions / filling behaviour

    • as mentioned by Paul an analysis of the timing patterns (between questions and time between filling two forms) could allow differentiating between a survey completed from recall vs. a survey completed while interviewing the respondent. I would expect somebody filling in the survey from recall to have a much more regular timing patterns than during a real interview, but my intuition may be wrong and it would probably be useful to generate some test datasets (interview / recall) and if we go for something more sophisticated you could probably use some machine learning algorithms to learn the difference between the 2 groups.
    • identify if all questions have not been answered in the same location when they should, or have been answered in the same location when they should not
    • place triggers about question sequence that would be suspicious (for instance it would be quite suspicious to go back to question A after having answered to question F, K and M)
    • model a standard filling behaviour based on collected submissions and detect outlier submissions (again I feel machine learning algorithms could be helpful here)
  • data collector

    • identify data collectors who present filling behaviours that deviate from what is defined as acceptable by the data collection requirements
    • identify data collectors who have very good command of the tools and seek feedback from them / ask them to train other data collectors
1 Like

@Thalie

Thanks for providing more details! Definitely this is what the tool could enable users achieve.

Paul