Commit Graph
249 Commits
Author SHA1 Message Date
vladkens 206f0942fe fix: continue pagination past empty pages (#265, #247) 2026-05-21 23:24:32 +03:00
vladkens c9aaad2e35 feat: add new fields to Tweet model (#279) 2026-05-21 22:11:22 +03:00
vladkens f8b9c95f97 test: expand cli coverage and fix ruff issues 2026-05-21 21:31:26 +03:00
59a658da1b feat: add add_cookie CLI command (#301)
Co-authored-by: Matvii Sakhnenko <[email protected]>
Co-authored-by: vladkens <[email protected]>
2026-05-21 21:05:56 +03:00
vladkens 14b3cd54c8 fix: robust gql pagination and mock data refactor 2026-05-21 20:27:41 +03:00
AmirHossein Khanzadehandvladkens 60b9b7acd0 Add API for getting tweets of a conversation (#252)
Co-authored-by: vladkens <[email protected]>
2026-05-21 17:25:24 +03:00
vladkens 304f9e3c09 ci: add pr-checks workflow 2026-05-21 16:04:45 +03:00
vladkensandterencedignon 8ba5b7638c feat: community scraping (closes #275)
Co-authored-by: terencedignon <[email protected]>
2026-05-21 15:51:50 +03:00
vladkens 0f2963dcc3 feat: add list_members API
Closes #254, co-authored by @cadenshokat
2026-05-21 14:00:25 +03:00
Terence Dignonandvladkens dcb2b93d68 feat: user about info (#277)
Co-authored-by: vladkens <[email protected]>
2026-05-21 13:31:17 +03:00
kediandvladkens 379c1019bd perf: parallel async downloads in update_gql_ops script (#264)
* Update _get_gql_ops.py

Make it way faster by keeping connection alive

* Update _get_gql_ops.py

---------

Co-authored-by: vladkens <[email protected]>
2026-05-21 13:02:45 +03:00
vladkens ff67298890 build: split lint/check and fix pyright example 2026-05-21 05:57:24 +03:00
vladkens 6e0db145e6 fix: drop user_by_id because X no longer supports it 2026-05-21 05:40:27 +03:00
vladkens b22da800cb chore: update mocked data & fix parser 2026-05-21 05:31:38 +03:00
vladkens 434fe43479 fix: improve proxy handling and xclid calculation 2026-05-21 05:12:10 +03:00
vladkens 4245a0747f build: update CI/release pipeline and add changelog 2026-05-21 05:10:54 +03:00
vladkens 68b2c03825 feat: pip -> uv 2026-05-21 02:23:01 +03:00
Marcinandvladkens 4b51422879 fix: restore scraping after X 2026-05 changes (issue #306) (#307)
* fix(api): refresh GraphQL queryIds (issue #306)

X rotated the GraphQL operation IDs around 2026-05; the previous IDs
return HTTP 404. Updated 13 OP_* constants from a fresh bundle.
Retweeters and Bookmarks were not present in the bundle, left as is.

* fix(xclid): fall back to XClientTransaction lib (issue #306)

Around 2026-04 X removed the e=>e+"."+ script-list pattern from its
bundle, which breaks get_scripts_list() and the native key derivation.

Adds an optional adapter that uses the upstream XClientTransaction
PyPI package (the same algorithm this module is ported from) when it
is installed. The native code path is preserved as a fallback so
existing setups are not affected.

To enable: pip install XClientTransaction

* fix(utils): handle X 2026 schema with legacy=null (issue #306)

X recently changed the GraphQL response shape for Tweet/User:

* legacy is now null (not a dict)
* Tweet fields previously inside legacy (full_text, entities,
  created_at, user_id_str, *_count) are on the top level
* User fields are split between top-level and the new sub-objects
  core (screen_name, name, created_at), avatar.image_url,
  location.location, privacy.protected, verification.verified,
  profile_bio.description
* Even when legacy is still a dict (e.g. nested User in
  core.user_results.result) it is partial, missing screen_name/name/
  created_at because those live in core.*

The old to_old_obj did **obj['legacy'], which raises TypeError on the
new schema. Replaced with a flatten that:

1. Merges the optional legacy with the top-level (top-level wins)
2. Pulls well-known fields from the new sub-objects up to the top level
3. Sets safe defaults for fields the parser dereferences directly

Verified end-to-end with api.search('python lang:en', limit=2):
24 tweets parsed successfully.

Backward compatible: when legacy is the old, complete dict, the merge
produces a flat dict equivalent to the previous output.

---------

Co-authored-by: vladkens <[email protected]>
2026-05-21 00:46:36 +03:00
Fla 5d7495f144 fix: properly close HTTP client in xclid, fix warning (#304) 2026-05-21 00:44:07 +03:00
Fla 7b67088972 Fix parsing of JS bundle to properly generate x-client-transaction-id (#303)
* Fix parsing of JS bundle to properly generate x-client-transaction-id, fixes #302

* Fix again xclid.py parsing after more Twitter changes
2026-05-21 00:42:42 +03:00
Vlad Pronsky 341c2f4701 doc: readme update 2025-04-29 04:28:50 +03:00
Vlad Pronsky 01aac62c17 v0.17.0 v0.17.0 2025-04-29 04:16:57 +03:00
Vlad Pronsky 440b342935 fix: deps 2025-04-29 04:05:27 +03:00
Vlad Pronsky 26ce624c05 chore: gql update 2025-04-29 03:57:06 +03:00
Vlad Pronsky e6116f438b fix: x-client-transaction-id #245 #248 2025-04-29 02:11:48 +03:00
Vlad Pronsky e4dce5d31f v0.16.0 v0.16.0 2025-03-09 18:13:25 +02:00
Vlad Pronsky 5488037c92 feat: trends #234 2025-03-07 05:33:46 +02:00
Vlad Pronsky 3e30c37499 chore: gql update 2025-03-07 03:58:28 +02:00
Vlad Pronsky 46bff105ef chore: drop deprecated api methods 2025-03-07 03:42:32 +02:00
Vlad Pronsky 5f10fd8ccc chore: lint fix 2025-03-07 03:07:08 +02:00
vladkens 0091e9ad86 Merge pull request #234 from mika-jpd/feature/scrape-explore-page-trends
feat: view explore page trends and search for posts in a trend
2025-03-07 01:52:12 +02:00
vladkens 072f838f2c Merge pull request #231 from viniciuskloppel/main
feat: allow searching for people
2025-03-07 01:50:23 +02:00
Mika Desblancs b5d1aeeb5f Added the ability to scrape trends in X explore by adding the two methods twscrape.api.list_explore and twscrape.api.list_explore_raw to prepare and query X's GenericTimelineById, parser methods twscrape.models.parse_trends and twscrape.models.parse_trend, a Timeline trend object and a twscrape.api.search_trend to search tweets in a trend with the appropriate search variables. 2025-02-25 17:31:56 -05:00
Vlad Pronsky 8b7c4235f3 chore: readme update 2025-02-16 00:42:11 +02:00
Vinicius Klöppel b7f338dc9f feat: enable searching for people 2025-01-19 13:14:19 -03:00
Vlad Pronsky 5ac141ebfc chore(release): v0.15.0 v0.15.0 2025-01-01 17:51:48 +02:00
Vlad Pronsky 1477fcd13f ci: add 3.13 to checks 2025-01-01 17:41:47 +02:00
Vlad Pronsky dbf989d58f fix: sqlite deprecated warnings 2025-01-01 17:40:44 +02:00
Vlad Pronsky c9bad5f399 doc: add section about limit param 2025-01-01 17:17:13 +02:00
Vlad Pronsky 1e49e9fab9 chore(gql): update endpoints to latest version 2025-01-01 17:16:38 +02:00
Vlad Pronsky f0379a637a chore(example): media download 2024-12-18 23:40:47 +02:00
Vlad Pronsky 1bdf36cff4 fix ci 2024-10-06 06:51:21 +03:00
Vlad Pronsky 8e84128c89 update ci deps v0.14 2024-10-06 06:45:18 +03:00
Vlad Pronsky 4728460827 update ci 2024-10-06 06:43:35 +03:00
Vlad Pronsky 359afb662b check py3.13 support 2024-10-06 06:28:55 +03:00
Vlad Pronsky ae75f9f5fb fix media type 2024-10-06 06:28:17 +03:00
Vlad Pronsky 3d8f2c69d0 maintenance 2024-10-06 06:03:24 +03:00
vladkens 74fbebb08e Merge pull request #218 from hponka/main
Update models.py
2024-10-06 05:29:25 +03:00
hponka b6bb72897b Update models.py
- added to tweet model: possibly_sensitive
2024-10-04 13:58:50 +03:00
Vlad Pronsky 664114e8bc add pinned tweets info to user profile #201 v0.13 2024-06-29 22:13:11 +03:00