Monitoring Gitlab Snippets for secrets with TruffleHog

Published: at 06:30 PM

Introduction

In the previous article, we explored how to monitor GitHub Gists for leaked secrets. In this article, we’ll implement a similar system, but with a focus on GitLab Snippets. We’ll configure TruffleHog to detect sensitive information, such as API keys and passwords, that might have been inadvertently exposed in public Snippets.

Table of contents

Open Table of contents

Snippets

GitLab Snippets are a feature in GitLab that allows users to share small pieces of code, configuration files, or any other text content. Similar to GitHub Gists, GitLab Snippets are a lightweight way to share code or text with others, whether for collaboration, documentation, or quick sharing.

Snippets API

The GitLab Snippets API allows developers to programmatically interact with GitLab Snippets, enabling the creation, retrieval, updating, and deletion of snippets through HTTP requests. With the API, users can automate the management of both personal and project snippets, access specific snippet content, and integrate snippet operations into their applications or workflows. The snippets API documentation can be found here. Similar to Github Gists you will need to create an access token.

TruffleHog

TruffleHog, developed by Truffle Security, is a powerful security tool designed to detect and prevent the leakage of sensitive information in code repositories. By scanning the entire commit history of Git repositories, TruffleHog identifies secrets such as API keys, passwords, and other confidential data using entropy analysis and regex matching. It integrates seamlessly into CI/CD pipelines, helping developers catch and eliminate potential security risks before they are pushed to production. Ideal for security audits and incident response, TruffleHog is an essential tool for safeguarding codebases against inadvertent exposure of sensitive credentials.

Monitoring script

We’ll create a simple shell script that queries the public Snippets API, retrieves each repository URL, and scans them with TruffleHog. This script is similar to the one used for monitoring Gists. First, we’ll define variables for file names, API tokens, and other essential configurations.

SNIPPETS_DOWNLOADED=downloaded-snippets.tmp
SNIPPETS_URLS=snippets-urls.tmp
PREV_SNIPPETS_URLS=downloaded-snippets-urls.log
NUM=100
TRUFFLE_LOG="trufflehog.log"
SNIPPETS_LOG="snippets.log"
TIMESTAMP=""
TOKEN="glpat-XXXXXXXXXXXXXXXXXXXX"

Next, we’ll continuously query the Snippets API in an infinite loop to extract the Git pull URL for each Snippet. These URLs will be stored in a sorted text file, with duplicates removed:

while true; do

  url="https://gitlab.com/api/v4/snippets/public?per_page=$NUM&created_after=$TIMESTAMP"
  echo "URL: $url"

  curl -L \
    -H "PRIVATE-TOKEN: $TOKEN" \
    -s \
    -o $SNIPPETS_DOWNLOADED $url

  TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

  cat $SNIPPETS_DOWNLOADED >> $SNIPPETS_LOG
  cat $SNIPPETS_DOWNLOADED | jq -r '.[].http_url_to_repo' | sort -u > $SNIPPETS_URLS

  echo "Downloaded $(wc -l $SNIPPETS_URLS)"

  cat $SNIPPETS_URLS >> $PREV_SNIPPETS_URLS
  sort -o $PREV_SNIPPETS_URLS -u $PREV_SNIPPETS_URLS

  rm $SNIPPETS_URLS
  rm $SNIPPETS_DOWNLOADED

  echo "[*] All snippets done, sleeping..."
  echo
  sleep 900
done

The final step is to run TruffleHog on each repository URL:

for snippet in $(cat $SNIPPETS_URLS); do
  if [[ $(grep -c "$snippet" $PREV_SNIPPETS_URLS) -eq 0 ]]; then
    echo "[+] Snippet: ${snippet}"
    trufflehog git --no-update --concurrency=3 -j $snippet --only-verified | tee -a $TRUFFLE_LOG
  fi
done

TruffleHog will download the Git repository and scan it for secrets. All detected secrets will be verified and logged to a file.

The complete script can be found below:

#!/bin/bash

SNIPPETS_DOWNLOADED=downloaded-snippets.tmp
SNIPPETS_URLS=snippets-urls.tmp
PREV_SNIPPETS_URLS=downloaded-snippets-urls.log
NUM=100
TRUFFLE_LOG="trufflehog.log"
SNIPPETS_LOG="snippets.log"
TIMESTAMP=""
TOKEN="glpat-XXXXXXXXXXXXXXXXXXXX"

while true; do
  url="https://gitlab.com/api/v4/snippets/public?per_page=$NUM&created_after=$TIMESTAMP"
  echo "URL: $url"

  curl -L \
    -H "PRIVATE-TOKEN: $TOKEN" \
    -s \
    -o $SNIPPETS_DOWNLOADED $url

  TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

  cat $SNIPPETS_DOWNLOADED >> $SNIPPETS_LOG
  cat $SNIPPETS_DOWNLOADED | jq -r '.[].http_url_to_repo' | sort -u > $SNIPPETS_URLS

  echo "Downloaded $(wc -l $SNIPPETS_URLS)"
  for snippet in $(cat $SNIPPETS_URLS); do
    if [[ $(grep -c "$snippet" $PREV_SNIPPETS_URLS) -eq 0 ]]; then
      echo "[+] Snippet: ${snippet}"
      trufflehog git --no-update --concurrency=3 -j $snippet --only-verified | tee -a $TRUFFLE_LOG
    fi
  done

  cat $SNIPPETS_URLS >> $PREV_SNIPPETS_URLS
  sort -o $PREV_SNIPPETS_URLS -u $PREV_SNIPPETS_URLS

  rm $SNIPPETS_URLS
  rm $SNIPPETS_DOWNLOADED

  echo "[*] All snippets done, sleeping..."
  echo
  sleep 900
done

Displaying results

TruffleHog saves its results in JSON format. To format the output and extract the most important information, you can use the jq command. An example of how to do this is provided below:

cat trufflehog.log | jq 'select(.Verified == true) | {DetectorName: .DetectorName, verified: .Verified, Raw: .Raw, RawV2: .RawV2, email: .SourceMetadata.Data.Git.email, repo: .SourceMetadata.Data.Git.repository}'

Conclusion

Utilizing TruffleHog allows you to automate the detection of sensitive information, such as API keys and passwords, that might have been unintentionally exposed in public Snippets.

References