/twitter-apify-fetcher — Library Skill: Twitter/X Data Fetcher

A standard agent-facing interface to the Apify Kaito Twitter scraper (kaitoeasyapi/twitter-x-data-tweet-scraper-pay-per-result-cheapest). Every agent fetches Twitter data through this skill instead of building its own Apify CLI call.

How It Is Invoked

Skill tool: twitter-apify-fetcher
Input: {
  mode: "search" | "profile" | "hashtag" | "list" | "conversation" | "tweet_ids",
  query: <string | string[]>,           // varies by mode (see below)
  max_items: <int>,                     // 5-1000, default 50
  query_type?: "Latest" | "Top",        // default "Latest"
  lang?: <ISO-639-1>,                   // "tr", "en", all if left empty
  min_likes?: <int>,                    // engagement filter
  min_retweets?: <int>,
  min_replies?: <int>,
  since?: "YYYY-MM-DD",                 // date range
  until?: "YYYY-MM-DD",
  blue_verified_only?: <bool>,
  has_media?: <bool>,
  output_path?: <abs path>              // dataset to stdout if absent
}

Mode Tables

Mode	query format	Example
search	string[] (Twitter advanced search syntax)	["AI startup TR", "fintech KOBİ"]
profile	string (handle, without @)	"elonmusk"
hashtag	string[] (with or without #)	["#yapayzeka", "kobi"]
list	string (Twitter list URL)	"https://x.com/i/lists/123..."
conversation	string (root tweet ID)	"1846987139428634858"
tweet_ids	string[] (ID array)	["18469...", "20493..."]

Credentials & Prerequisites

source ~/.claude/.env  # APIFY_API_TOKEN comes from here
which apify || brew install apify-cli
[ -f ~/.apify/auth.json ] || apify login -t "$APIFY_API_TOKEN"

If APIFY_API_TOKEN is empty → FAIL fast: tell the user "credentials/.env missing".

Flow

1. Input Validation

• mode is required, must be in the list
• query must be the correct type for the mode (search → array, profile → string, etc.)
• max_items between 5-1000 → clamp + WARN if outside
• since/until format YYYY-MM-DD → FAIL otherwise

2. Payload Build

Generate the Apify input JSON by mode:

// search mode example
{
  "searchTerms": ["AI startup TR since:2026-01-01 until:2026-04-29"],
  "maxItems": 100,
  "queryType": "Latest",
  "lang": "tr",
  "min_faves": 10
}

// profile mode example
{
  "from": "elonmusk",
  "maxItems": 50,
  "queryType": "Latest"
}

If date filters are present, inject them into searchTerms: "... since:2026-01-01 until:2026-04-29".

3. Invocation (file-based, hung-pipe-safe)

INPUT_FILE=$(mktemp -t apify-twitter-in.XXXXXX.json)
OUTPUT_FILE=${output_path:-$(mktemp -t apify-twitter-out.XXXXXX.json)}
echo "$PAYLOAD_JSON" > "$INPUT_FILE"

# 180-second watchdog (macOS/Linux portable; no coreutils required):
apify call kaitoeasyapi/twitter-x-data-tweet-scraper-pay-per-result-cheapest \
  --silent --input-file "$INPUT_FILE" --output-dataset > "$OUTPUT_FILE" &
APIFY_PID=$!
( sleep 180 && kill -TERM $APIFY_PID 2>/dev/null ) & WATCHDOG=$!
wait $APIFY_PID; RC=$?
kill $WATCHDOG 2>/dev/null

# Not an RC check — a file-integrity check (apify-CLI can hang on post-call cleanup
# under Node 24+; the watchdog cuts it with SIGTERM, RC becomes 143, but the dataset
# has already been written):
if [ ! -s "$OUTPUT_FILE" ]; then
  echo "apify failed: output file empty (rc=$RC, watchdog likely fired before any data written)" >&2
  exit 1
fi
if ! jq -e '. | length > 0' "$OUTPUT_FILE" >/dev/null 2>&1; then
  echo "apify failed: output not valid JSON or zero items (rc=$RC)" >&2
  exit 1
fi
if [ $RC -ne 0 ]; then
  # Data is sound but the CLI could not exit cleanly — Node 25 post-call hung pattern. Accepted.
  echo "apify CLI rc=$RC but output valid; Node post-call cleanup hung — continuing." >&2
fi

# parse from the file:
jq '. | length' "$OUTPUT_FILE"           # tweet count
jq '.[0:5]' "$OUTPUT_FILE"               # view the first 5 items

⚠️ NEVER pipe the output of apify call. Redirecting it to | head, | tail, | grep, | jq, | awk, or any other command is forbidden. The Apify-CLI Node stream does not handle SIGPIPE properly (especially Node 24+); if the pipe receiver closes, the CLI hangs for 10+ minutes, exhausts the parent worker timeout, and blows up the council cycle (evidence: 2026-05-02 distribution-strategy-synthesis-hub cycle 1, 15 min timeout, twinkling-wandering-volcano.md). First write to a file with > "$OUTPUT_FILE", then read from the file. If the output is very large, use the output_path parameter and process the file with something like jq -c '.[0:50]' "$OUTPUT_FILE".

Watchdog note: GNU timeout or gtimeout is not default on macOS — the sleep + kill pattern above works on every POSIX system. If you install gtimeout via brew install coreutils, you can collapse it to a single line.

Timeout: 180 sec (large queries can take 60-120 sec; the bash watchdog enforces it).

4. Mock-tweet Filter (CRITICAL)

When Kaito returns 0 results, it returns a "mock_tweet" for the minimum charge. Always filter it out:

const real = results.filter(r => r.type === "tweet" && r.id);
const mocks = results.filter(r => r.type === "mock_tweet");
if (real.length === 0) {
  return { status: "no_results", warning: "Apify returned a mock-tweet, revise the query", mock_count: mocks.length };
}

5. Output Normalization

For each real tweet, at minimum these fields:

{
  "id": "...",
  "url": "https://x.com/...",
  "text": "...",
  "created_at": "ISO 8601",
  "lang": "tr",
  "engagement": { "likes": 1308, "retweets": 63, "replies": 182, "quotes": 3, "views": 27698, "bookmarks": 13 },
  "author": { "username": "...", "id": "...", "name": "...", "followers": N, "verified": bool, "blue_verified": bool },
  "is_reply": bool,
  "is_retweet": bool,
  "is_quote": bool,
  "media": [...],
  "hashtags": [...],
  "mentions": [...],
  "urls": [...]
}

If you want to keep the full Apify response, provide output_path and the raw JSON is written there.

Cost Estimate

• Every 1000 tweets ≈ $0.25
• max_items: 100 → ~$0.025
• A mock-tweet also incurs the minimum charge (~$0.001-0.0