Keylime: Fixing Push Attestation Failures With Full IMA Logs

by Omar Yusuf 61 views

Introduction

Hey guys, let's dive into a tricky issue we've encountered with Keylime's push attestation mechanism. Specifically, we're seeing attestation failures because the agent is sending the full Integrity Measurement Architecture (IMA) log every time, instead of just the updates. This can lead to PCR mismatches and, ultimately, failed attestations. So, if you're dealing with Keylime and push attestation, this is something you'll definitely want to understand. This article provides a comprehensive overview of the issue, its root cause, and potential solutions. We'll also explore a reproducer to help you verify the problem in your own environment. The goal here is to ensure that your Keylime setup is robust and reliable, providing the security guarantees you expect. Understanding the intricacies of IMA logs and their interaction with the Keylime verifier is crucial for maintaining a secure and trusted environment. So, let's get started and unravel this issue together!

Problem Description

The core of the problem lies in how Keylime handles IMA logs in the push model. In Keylime, the running_hash value plays a critical role in verifying the integrity of the system. Initially, this running_hash is derived from the last quote, or a zeroed value if there's no prior quote. The Keylime verifier expects each quote to contain only the new IMA log entries generated since the previous attestation. This incremental approach is designed to optimize performance and reduce the amount of data transmitted. However, the rust-keylime agent, in its current implementation, sends the complete IMA log with every push attestation. This means that some log lines are processed multiple times by the verifier. Since the verifier initializes the running_hash based on the previous query, re-processing log lines leads to a mismatch in the calculated PCR values, causing the attestation to fail. This issue highlights a critical discrepancy between the expected behavior of the Keylime verifier and the actual behavior of the rust-keylime agent in the push model. Addressing this inconsistency is crucial for the proper functioning of Keylime's attestation process.

Here's a breakdown of what's happening:

  1. Initial running_hash: The Keylime verifier gets the initial running_hash from the last quote or starts with a zeroed value.
  2. Expected Behavior: Keylime expects each quote to contain only the new IMA log entries.
  3. Actual Behavior: The rust-keylime agent sends the complete IMA log.
  4. Result: The verifier reprocesses log lines, causing a PCR mismatch and attestation failure.

Verifying the Issue with Logs

To confirm this, I added some extra logging to the keylime_verifier. Check out these log snippets:

# initial attestation
Aug 13 06:41:00 df5301b0-68a2-42bd-89da-dd72bdca2247 keylime_verifier[18816]: 2025-08-13 06:41:00.049 - keylime.ima - ERROR - Initial running hash: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Aug 13 06:41:00 df5301b0-68a2-42bd-89da-dd72bdca2247 keylime_verifier[18816]: 2025-08-13 06:41:00.052 - keylime.ima - DEBUG - IMA: ignoring excluded path /usr/bin/kmod

# subsequent attestation
Aug 13 06:42:00 df5301b0-68a2-42bd-89da-dd72bdca2247 keylime_verifier[18816]: 2025-08-13 06:42:00.261 - keylime.ima - ERROR - Initial running hash: b'y\xc0N\xd3\xc9\xc2o\x1e\x87Kh\xe5\xfb\x8f\x19{\xa6\xb8\xb1:\x82P9VJ\xd1&\r\t\x9e%\xad'
Aug 13 06:42:00 df5301b0-68a2-42bd-89da-dd72bdca2247 keylime_verifier[18816]: 2025-08-13 06:42:00.262 - keylime.ima - DEBUG - IMA: ignoring excluded path /usr/bin/kmod

Notice how the same file (/usr/bin/kmod) is being processed in both attestations. This clearly indicates that the verifier is re-evaluating entries that should have already been accounted for, leading to the mismatch. The initial running hash changes between the first and subsequent attestation, further confirming that the verifier is not receiving incremental updates as expected. This logging evidence provides concrete proof of the issue, highlighting the need for a solution that ensures only new IMA log entries are processed during each attestation.

Potential Solutions

So, what can we do about this? There are a couple of ways we might tackle this issue:

  1. Verifier Modification: One option is to modify the verifier to handle quotes containing the full IMA log, especially if the quote indicates that the log is complete. This could involve initializing the running_hash with a zeroed value when a complete log is received.
  2. Agent Modification: Alternatively, the agent could be changed to send only the updated part of the IMA log. This aligns with Keylime's expected behavior and would likely be the more efficient solution.
  3. Hybrid Approach: Ideally, we could implement both solutions. The verifier could be made more robust by handling both full and incremental logs, while the agent is optimized to send only the necessary updates. This would provide the best of both worlds: compatibility and efficiency.

Each of these approaches has its own set of trade-offs. Modifying the verifier might add complexity to the verification process, while modifying the agent would require changes to the rust-keylime codebase. A hybrid approach, while potentially the most robust, would also involve the most development effort. The choice of solution will likely depend on factors such as the desired level of compatibility, performance considerations, and the resources available for development and testing. It's crucial to carefully evaluate each option to determine the best path forward for resolving this issue.

Reproducing the Issue

To reproduce this issue, you can follow these steps:

  1. Configure push attestation on a test system so that the initial attestation passes.
  2. Execute a file that is allowed by the policy. This will extend the IMA log without causing an immediate attestation failure.
  3. Observe the subsequent attestation failure.

I've also created a tmt based reproducer that leverages test code from various projects. Here's how you can use it:

$ git clone -b test_push_attestation https://github.com/ansasaki/keylime-tests.git
$ cd keylime-tests
$ cat > plans/reproducer.fmf <<_EOF
summary:
  Tests used by Packit/TFT CI on Github to test upstream keylime

environment+:
  TPM_BINARY_MEASUREMENTS: /var/tmp/binary_bios_measurements
  KEYLIME_RUST_CODE_COVERAGE: 0

discover:
  how: fmf
  test:
   - /setup/apply_workarounds
   - /setup/configure_tpm_emulator
   - /setup/install_upstream_keylime
   - /setup/install_upstream_rust_keylime
   - /setup/enable_keylime_debug_messages
   - /setup/configure_kernel_ima_module/ima_policy_simple
   - /functional/push-attestation-on-localhost

execute:
    how: tmt
_EOF

$ ### have a Fedora42 test system (no TPM required, will be emulated) ready and available through $IP
$  tmt -c distro=fedora-42 run -vvv discover plan -n plans/reproducer provision -h connect -g $IP prepare execute
# observe the test failure

                00:04:47 fail /functional/push-attestation-on-localhost (on default-0) [7/7]
                    ..:..:.. pass /Setup-push-attestation-environment (subresult)
                    ..:..:.. pass /Add-keylime-agent (subresult)
                    ..:..:.. fail /Running-allowed-scripts-should-not-affect-attestation (subresult)
                    ..:..:.. fail /Fail-keylime-agent (subresult)
                    ..:..:.. pass /Cleanup-push-attestation-test (subresult)

Make sure you have a Fedora 42 test system available (TPM is not required as it will be emulated). This reproducer will help you quickly verify the issue in a controlled environment. By running this test, you can confirm that the full IMA log transmission is indeed causing attestation failures, providing further evidence for the need to address this problem.

Code Versions

For reference, here are the code versions used during the investigation:

These versions can be helpful for anyone trying to replicate the issue or understand the context in which it was discovered. Keeping track of code versions is crucial for effective debugging and ensuring that fixes are applied to the correct codebase. By providing this information, we can facilitate collaboration and make it easier for others to contribute to the resolution of this problem.

Conclusion

In conclusion, the issue of the rust-keylime agent sending the full IMA log in push attestation mode is a significant one, leading to attestation failures due to PCR mismatches. We've seen how the Keylime verifier expects incremental updates to the IMA log, while the agent's current behavior doesn't align with this expectation. Guys, addressing this discrepancy is crucial for the reliability of Keylime's push attestation mechanism. We've explored potential solutions, including modifying the verifier, the agent, or a combination of both. The provided reproducer offers a practical way to verify the issue and test any proposed fixes. By understanding the root cause and potential solutions, we can work towards a more robust and efficient Keylime implementation. Whether it's tweaking the verifier to handle full logs or optimizing the agent to send only the necessary updates, the goal is to ensure that Keylime continues to provide the strong security guarantees we rely on. Let's collaborate to resolve this and keep Keylime rock solid!