suKTech24

Encoding & Decoding 101

By suktech24, Sun 29 Sep 2024, Estimated reading time : 9-10 mins

This blog post will explore the basic of encoding, decoding, various encoding schemes, an example of cybersecurity attack related to encoding and decoding and python code that illustrate the concepts.

  1. What is Encoding & Decoding?
  2. CIA Triad and Encoding Myth
  3. Encoding Schemes Examples
    • Base64 Encoding
    • Hexadecimal Encoding
    • HTML Encoding
    • Protobuf Encoding
    • URL Encoding
  4. CyberChef
  5. CVE-2022-30619
  6. Python code – Base64 Encoding and Decoding

1. What is Encoding & Decoding?

Encoding data is a process involving changing data into a new format using a scheme. Encoding is a reversible process and data can be encoded to a new format and decoded to its original format. Encoding typically involves a publicly available scheme that is easily reversed.

Encoding data is typically used to ensure the integrity and usability of data and is commonly used when data cannot be transferred in its current format between systems or applications.

Encoding is not used to protect or secure data because it is easy to reverse.

Encoding converts data into a different format using a scheme that can be easily reversed. Examples include Base64 encoding, which encodes binary data into ASCII characters, making it easier to transmit data over media that are designed to deal with textual data.

2. CIA Triad and Encoding Myth

When I first started studying, I thought encoding/decoding supports the integrity from CIA triad. But I was wrong since data encoded can be decoded using specific encoding schemes and it is a reversible process. Anyone can decode as long as they use the appropriate encoding scheme. The level of protection and security encoding provides is not the same level as encryption.

The three letters in “CIA triad” stand for Confidentiality, Integrity, and Availability. The CIA triad is a common model that forms the basis for the development of security systems. They are used for finding vulnerabilities and methods for creating solutions.

Sourced from, SANGFOR, https://www.sangfor.com/glossary/cybersecurity/what-is-cia-triad

3. Encoding Schemes Examples

  1. Base64 Encoding
    • This is a commonly used technique for encoding binary data into ASCII text format and is often used for encoding data such as images, audio, and video files over a network or for storage in a database.
  2. Hexadecimal Encoding
    • This encoding technique represents binary data in a hexadecimal (base 16) format. Therefore, it is commonly used for representing binary data in a more human-readable form and is often used for debugging and troubleshooting purposes.
  3. HTML Encoding
    • HTML encoding replaces certain characters that are semantically meaningful in HTML markup, with equivalent characters that can be displayed to the user without affecting parsing the markup. There are various characters that are part of the HTML markup itself (such as <, > etc.). To use these within the document as content you need to HTML encode them such as &lt; for the less than < sign and &gt; for > sign
    • HTML encoding safeguards web data by converting special characters into HTML entities, preventing interpretation errors and enhancing security against XSS attacks. Decoding reverses this process, ensuring accurate interpretation of encoded data, vital for seamless communication and content manipulation across diverse platforms and browsers.
  4. Protobuf Encoding
    • Protobuf, which is short for “Protocol Buffers,” is an efficient, language-agnostic data serialization mechanism. It enables developers to define structured data in a .proto file, which is then used to generate source code that can write and read data from different data streams.
    • Protobuf uses a binary data format, which is more compact and faster to read and write than text-based formats. It allows efficient serialization and deserialization of structured data, making it faster and more compact than JSON or XML. It also provides an interface definition language (IDL) that makes it easy to define the structure of the data to be serialized. E.g. Python, C++, Java, etc., https://protobuf.dev/getting-started/
    • For each field, the protocol buffer will encode a key and a value. 
    • https://protobuf.dev/programming-guides/encoding/
  5. URL Encoding
    • URL encoding is a process where special characters within a URL are converted into a format that can be transmitted safely over the internet. Without URL encoding, certain characters in the URL, such as spaces or special symbols, can cause errors or be misinterpreted by web servers and browsers. This can lead to broken links or security vulnerabilities.
    • URL encoding, also known as percent encoding, involves replacing special characters with a percent sign followed by two hexadecimal digits, representing the ASCII code for that character. For example, a space is encoded as ‘%20’ while an asterisk is encoded as ‘%2A’.
    • URL encoding – URL encoding, also known as percent encoding, involves replacing special characters with a percent sign followed by two hexadecimal digits, representing the ASCII code for that character. For example, a space is encoded as ‘%20’ while an asterisk is encoded as ‘%2A’.
    • URL encoding four steps
      • Identify the special characters within the URL
      • Convert each special character into its corresponding percent-encoded value
      • Replace the special characters in the URL with their respective percent-encoded values
      • Verify the encoded URL to ensure accuracy

4. CyberChef

I recommend you to check out CyberChef website and experiment with it for above encoding schemes. According to their github repo website, https://github.com/gchq/CyberChef, CyberChef is a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser.

CyberChef makes it simple for users to carry out both simple and complex data manipulation tasks within a web browser such as:

  • Decode encoded data, such as base64 or XOR
  • Perform data conversions, such as timezones
  • Decrypt and disassemble shellcode
  • Compress and decompress data
  • Calculate hashes and checksums

You can also create and save recipes for later usage.

You select the what you want under Operations, bring it under Recipe and enter the relevant information. Below illustrates 5 encoding schemes from Examples of Encoding section.

  1. Base64 Encoding
  2. Hexadecimal Encoding
  3. HTML Encoding
  4. Protobuf (Protocol Buffers) Encoding
  5. URL Encoding

CVE related to encoding

Out of 5 encoding schemes, Base64 encoding CVE is discussed.

CVE-2022-30619

  • The vulnerability involves editable SQL queries behind Base64 encoding being sent from Client-Side to Server-Side via a specific API used in the legacy Work Center module. The attack can be executed by any authenticated user under any rule.
  • The vulnerability allows attackers to perform SQL injection by manipulating encoded SQL queries within the Agile Point NX environment.
  • https://www.clouddefense.ai/cve/2022/CVE-2022-30619
  • https://www.cve.org/CVERecord?id=CVE-2022-30619
  • SQL Simple example
    • Original SQL
      • `SELECT * FROM users WHERE username=’admin’ AND password = ‘input’`
    • Malicious SQL
      • `SELECT * FROM users WHERE username=’admin’ –‘ AND password= ‘anything’`
    • Base64 encoding Malicious Query
      • U0VMRUNUICogRlJPTSB1c2VycyBXSEVSRSB1c2VybmFtZSA9ICdhZG1pbicgLS0nIEFORCBwYXNzd29yZCA9ICdhbnl0aGluZyc=

Python code – Base64 Encoding and Decoding

import base64

def encode_sql(sql_query):
    return base64.b64encode(sql_query.encode()).decode()

def decode_sql(encoded_query):
    return base64.b64decode(encoded_query.encode()).decode()

def simulate_query_execution(sql_query):
    # print(f"Executing query: {sql_query}")
    if "username= 'admin'" in sql_query and "password = 'input'" in sql_query:
        return "Result: Admin Access granted. Correct username and password."
    elif "username= 'admin'" in sql_query and "--" in sql_query:
        # SQL injection detected
        return "Result: Admin Access granted! (SQL Injection successful)"
    else:
        return "Result: Access denied. Incorrect username or password."

def process_queries(queries):
    results = []
    for query_type, sql in queries.items():
        encoded = encode_sql(sql)
        decoded = decode_sql(encoded)
        execution_result = simulate_query_execution(decoded)
        
        results.append(f"""
            {query_type} SQL Query: {sql}
            Base64 Encoded SQL : {encoded}
            {execution_result}
            """
        )
    return "\n".join(results)

def main():
    queries = {
        "Original": "SELECT * FROM users WHERE username= 'admin' AND password = 'input'",
        "Malicious": "SELECT * FROM users WHERE username= 'admin' --' password = 'anything123'",
        "Testing": "SELECT * FROM users WHERE username= 'user1' AND password = 'input'"
    }
    
    results = process_queries(queries)
    print(results)

if __name__ == "__main__":
    main()

When you execute above python script, you will see below result.

Python Encoding Steps

  • base64.b64encode(sql_query.encode()).decode()
  • Step 1 : Encoding string into byte –> Step 2 : then to Base64 –> Step 3: Decoding to string
  • Step 1 : sql_query.encode()
  • Step 2 : Base64 encode
  • Step 3 : Decode to string
    • base64_string = base64_bytes.decode()
  • Step 1 + 2 + 3 : Combing all of them together
    • def encode_sql(sql_query):
      return base64.b64encode(sql_query.encode()).decode()
import base64

def demonstrate_base64_encoding(sql_query):

    # Step 1 : Encode to bytes
    bytes_sql = sql_query.encode()
    
    # Step 2 : Base64 encode
    base64_bytes = base64.b64encode(bytes_sql)
    
    # Step 3 : Decode to string 
    base64_string = base64_bytes.decode()
    
    # All Step 1 + 2 + 3
    result = base64.b64encode(sql_query.encode()).decode()
    
    return (f"""
    Original SQL query : {sql_query}
    Step 1 - Encoded string to bytes : {bytes_sql}
    
    Step 2 = Base64 Encoded bytes : {base64_bytes}
    
    Step 3 - Decoded Base64 to string : {base64_string}
    
    All Step 1 + 2 + 3 : {result}
    """)

def main():
    sql_query = original_sql_query = "SELECT * FROM users WHERE username= 'admin' AND password = 'input'"
    output = demonstrate_base64_encoding(sql_query)
    print(output)

if __name__ == "__main__":
    main()

If you execute the code, you will see below output.

Summary

The summary for this blog post is below:

  • Encoding – Encoding data is a process involving changing data into a new format using a scheme. Encoding is a reversible process and data can be encoded to a new format and decoded to its original format. Encoding typically involves a publicly available scheme that is easily reversed.
  • CIA triad – The CIA (Confidentiality, Integrity, and Availability) triad is a common model that forms the basis for the development of security systems.
  • Encoding schemes
    • Base64 Encoding – Encode binary data into ASCII text format
    • Hexadecimal Encoding – Encode binary data in a hexadecimal (base 16) format
    • HTML Encoding – HTML encoding replaces certain characters that are semantically meaningful in HTML markup, with equivalent characters that can be displayed to the user without affecting parsing the markup.
    • Protobuf Encoding – For each field, the protocol buffer will encode a key and a value. Protocol buffers are binary, hence, compared to JSON(just a text), it is more performant and faster.
    • URL Encoding – URL encoding is a process where special characters within a URL are converted into a format that can be transmitted safely over the internet. Without URL encoding, certain characters in the URL, such as spaces or special symbols, can cause errors or be misinterpreted by web servers and browsers. This can lead to broken links or security vulnerabilities.
  • CyberChef – CyberChef is a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser. These operations include simple encoding like XOR and Base64, more complex encryption like AES, DES and Blowfish, creating binary and hexdumps, compression and decompression of data, calculating hashes and checksums, IPv6 and X.509 parsing, changing character encodings, and much more.
  • Base64 Encoding CVE example – CVE-2022-30619
  • Python code for encoding and decoding
    • import base64
      def encode_sql(sql_query):
      return base64.b64encode(sql_query.encode()).decode()
      def decode_sql(encoded_query):
      return base64.b64decode(encoded_query.encode()).decode()

References

They are embedded in the blog post.

Leave a comment