Skip to main content
Meet NetWitness at RSA Conference 2024!
Stop by our booth #254 or book a meeting with an expert. Reserve Your Spot Today!
Industry Perspectives

Decrypting the Benefits of Proactive Threat Hunting in Encrypted Network Traffic

  • by Cody Spooner

How a proactive approach to threat hunting can help organizations stay ahead of misconfigurations and cyberattacks.

Encrypted network traffic has been on the rise for years due to a combination of factors, including the proliferation of data privacy and security, the adoption of cloud-based services, and changes in how people work and communicate. But encrypted network traffic also presents unique challenges for threat hunters–by obscuring network traffic, encryption makes it more challenging to detect potential threats. 

In many ways, encrypted traffic is like Schrodinger’s cat of information security–we think the data is encrypted, and we hope it’s encrypted. Still, we don’t know unless we put our eyes on it. Ultimately, security analysts need to validate whether encryption is working as intended to have 100% certainty that data is moving safely and securely. 

In this context, threat hunting is more about identifying risk than anything else. Although we often glorify threat hunting as finding the “bad guys,” it’s more about finding misconfigurations–which organizations are far more likely to see than threat actors or malicious activity. 

Ultimately, by understanding the challenges of encrypted traffic and implementing proactive threat-hunting strategies, organizations can better protect themselves from the risk of a misconfiguration or a cyberattack. 


A brief explanation of encryption

Network encryption is converting data into code or cipher to secure it while transmitting it over a network. Encryption protects sensitive data from unauthorized access, interception, and eavesdropping. It involves converting plaintext data into ciphertext–a scrambled or unreadable format. The sender transmits ciphertext over the network, and the receiver decrypts it using a decryption key to convert it back to plaintext. 

With plaintext data, you essentially have a smoking gun or evidence that encrypted traffic isn’t adequately secured due to a misconfiguration or cyberattack. But with encrypted data, organizations need to change their approach to one that’s more behavior-based and focused on making inferences based on the available artifacts. 


Checking facts and identifying risks

Checking facts and identifying risks is all about challenging the status quo–looking at data, recognizing what we expect that data to look like, and making sure it meets our expectations.

If your organization has a business policy that states “X, Y, and Z are not allowed,” it might be tempting to take that at face value. However, further exploration is almost always warranted. Is that policy being adhered to, or are we violating that policy? If so, is there a risk to the business? What is that risk?

That’s where threat hunting comes into play. By proactively identifying that risk, we can ensure that an organization follows all of the procedures put into place.

Tips for checking facts and identifying risks

  • Begin with existing policies. Look at the policies you have in place today and determine the required datasets to challenge them. 
  • Start with a hypothesis. Threat hunting often begins with a hypothesis. Narrow your scope to what the threat is and what you want to find before you start looking for it. Then, dive into your toolset and look through your data to understand an expected day on your network. Ultimately, proactive threat hunting comes down to understanding your environment and what the day-to-day behavior looks like. By pairing situational awareness with strong data enrichment, you can efficiently identify anomalies within your dataset and easily test your hypothesis.
  • Use protocol mismatches to inspect data passing through the network. Protocol mismatches occur when network traffic uses a different protocol than the recipient expects. To detect protocol mismatches, use network traffic analysis tools to inspect network traffic for inconsistencies. Once a protocol mismatch is detected, determine if it warrants further investigation–if a misconfiguration causes the mismatch, it may not require further investigation; if malicious activity causes the mismatch, take additional steps to identify the attacker and mitigate the risk. 
  • Examine client metakeys. The client meta key provides insight into the client application (user-agent field in the packet) used to initiate the observed traffic. When traffic is encrypted, the user agent is generally not visible. When a web server redirects HTTP to HTTPS, the user-agent string is often ‘leaked.’ The user agent field is also populated when an HTTP/S protocol mismatch occurs. If the traffic is being decrypted, the client metakey provides visibility to threat hunters into the client application for HTTP and HTTPS traffic.
  • Look at the cipher key. The cipher key provides insight into the algorithm used for the data encryption. Examining the cipher key for out-of-date or weak algorithms provides valuable insight into the potential risk of encrypted traffic. This exercise has become more frequent in recent customer audits for compliance and cyber insurance.   
  • Consider the version key. The version key provides the cryptographic protocol version used for the network session (such as “TLS 1.0”). An attacker may exploit a vulnerability in an older version of a cryptographic protocol to identify potential risks, monitor network traffic for the use of outdated or unauthorized versions of cryptographic protocols. 


Supporting audits

With increased audits from cyber insurers and various compliance regulations, organizations must provide insight into how the sensitive data traversing the network is being secured. This includes analyzing the cryptographic protocols and the encryption algorithms (ciphers). 

Many organizations wonder, “How do we translate this data to our business?” 

Tips for supporting audits

  • Choose a tool with reporting capabilities. It’s essential that whichever tool you leverage can translate all of the data it ingests back to your business needs. Sometimes, that includes presenting that information to executive leadership in ways they understand. 
  • Translate data for actionable results. Most businesses need to break down what their traffic looks like in the network environment. It’s important from a business standpoint that that translation takes place so that the SOC can provide the answers needed for the business to make new policies and procedures. 


Jittery robots and C2 traffic

Command and control (C2) traffic is when an adversary attempts to communicate with compromised systems and control them. Generally, the command server awaits some form of beacon–if a threat actor can get a malicious payload past the first line of defense, we have the start of a C2 infection.

The implant gets executed, the client side of the framework comes to life, and it starts beaconing back out to the command server. But that beacon interval will change based on the framework the threat actor uses–it’s all very customizable, making the lives of analysts and threat hunters more difficult. 

Historically, threat hunters could quickly identify patterns. For instance, if they could see a connection beaconing out every 30 seconds to an unknown device–especially in plaintext–that would indicate a C2 attack. 

Today, along with the increase in encrypted traffic, the concept of “jitter” has been added to most C2 frameworks. The term “jitter” refers to irregularity. In the context of network encryption, “jitter” is the introduction of randomness to C2 traffic. For example, that could be randomness in the beacon time or the payload size. 

The introduction of jitter into C2 beacons makes identifying suspicious traffic patterns more difficult. It also makes detection rules that rely on set patterns in recurring artifacts, such as the beacon’s callout timestamp, obsolete. Technically, we can still detect that behavior, but it often needs a human touch. This is a use case where proactive threat hunting becomes exceptionally valuable. From an automated detection standpoint, using baselines and time windows can effectively identify C2 behavior. The results of an effective threat hunt can help tune threat detection content to your environment.

If you span out an investigation long enough–say, instead of looking at 30 minutes, looking at a few hours to a couple of days–you start to see that even the randomness has somewhat of a pattern. Again, jitter is customizable, but it doesn’t tend to stray too far away from the primary value. 

One noteworthy piece of advice: identifying behaviors in encrypted network traffic may be critical, but it is not the end-all of an investigation–it’s a good starting point for finding those behaviors, so you can dive deeper into an investigation.

Tips for investigating C2 traffic

  • Maintain situational awareness and clear-cut visibility. Identify where your blind spots are, cover as many as possible, and attempt to understand how your data flows through the network. When you have enough unanswerable questions due to unknowns, you can start to piece the picture together to tell a holistic story. By the time you get to the end of that story from your investigation, you can make a more educated decision as to “Do I need to pursue this, or can we drop the investigation?”
  • Choose a tool to help. No matter what tool you use, it’s crucial to organize your data during an investigation in a way that’s meaningful to you. Every SOC, along with every analyst, has a slightly different workflow for how data is investigated. Your tool of choice must be flexible enough to support your workflow.  


JA3 and JA3S

JA3 and JA3S are open-source concepts for fingerprinting the client application in Transport Layer Security (TLS)  traffic. 

JA3 creates a unique fingerprint of a TLS client hello message sent by a client when establishing a TLS or SSL connection with a server. JA3S is a variant of JA3 specifically designed to fingerprint TLS server hello messages sent to a server in response to a client hello message. Together, they create a relatively unique fingerprint of that connection. 

Since they are hash values, creating a plan to leverage them effectively is important. Hash values aren’t human-readable (they look like random characters put together into a string). However, they can be valuable when utilized effectively. 

Tips for leveraging JA3 and JA3S

  • Provide hash values as indicators of compromise (IoCs). Like any other IoC, you can feed JA3 and JA3S hashes into a tool of your choice, tag them, and alert on them. If you leverage JA3/s from a threat intel perspective, make sure you monitor, maintain, and prune them, just like any other IoC artifact. A SOAR and TIP platform is very useful for the automated curation of IoC artifacts.
  • Baseline JA3 and JA3S. Set a time window and capture the unique pairs of JA3 and JA3S to gather critical servers and alert on new anomalies. JA3S values don’t change as often as JA3 values, so if you combine the hashes, monitor critical servers, and baseline them; anomaly detection will be more high-fidelity.  
  • Leverage a user entity and behavior analytics (UEBA) tool. A UEBA tool can allow organizations to conduct a baseline over a more extended period for more devices. Ultimately, UEBA is all about anomaly detection. An effective UEBA will continue to learn and alert on outliers in the dataset. 


How NetWitness can Help

A network detection and response (NDR) platform like NetWitness provides various tools and capabilities for threat hunting and analysis of encrypted network traffic. With NetWitness, organizations can:

  • Decrypt and analyze encrypted network traffic so that security teams can monitor and analyze network traffic, allowing them to detect potential security threats concealed within encrypted traffic. 
  • Identify network activity patterns, detect potential anomalies and suspicious behavior, and identify security threats in real-time using a range of tools and capabilities for threat hunting, including advanced analytics and machine learning algorithms. 
  • Quickly identify and respond to potential threats with visualization and reporting capabilities, including dashboards, alerts, and reports that provide real-time information on network activity and potential security threats. 


Learn more about how NetWitness can help your organization adopt a proactive approach to threat hunting and reduce the risks associated with encrypted network traffic, and schedule a demo today.