feat: scrape user profile, hashtag, search or query or advanced search, latest tweets, or top tweets

This commit is contained in:
Jarrian
2023-09-09 21:10:44 +08:00
parent 07ece4e29a
commit ed0df7a0a2
3 changed files with 262 additions and 21 deletions

View File

@@ -18,6 +18,12 @@ TWITTER_PASSWORD=# Your Twitter Password
## Usage
- Show Help
```bash
python scraper --help
```
- Basic usage
```bash
@@ -27,15 +33,90 @@ python scraper
- Setting maximum number of tweets. defaults to `50`.
```bash
python scraper --tweets=500 # Scrape 500 Tweets
python scraper --tweets=500 # Scrape 500 Tweets
```
### Options and Arguments
- Options and Arguments
```bash
usage: python scraper [arg]
usage: python scraper [option] ... [arg] ...
Arguments Description
--tweets : No. of tweets. default: 50.
e.g. --tweets=500
options: description
-t, --tweets : Number of tweets to scrape (default: 50).
e.g.
-t 500
--tweets=500
-u, --username : Twitter username.
Scrape tweets from a user's profile.
e.g.
-u elonmusk
--username=@elonmusk
-ht, --hashtag : Twitter hashtag.
Scrape tweets from a hashtag.
e.g.
-ht javascript
--hashtag=javascript
-q, --query : Twitter query or search.
Scrape tweets from a query or search.
e.g.
-q "Philippine Marites"
--query="Jak Roberto anti selos"
--latest : Twitter latest tweets (default: True).
Note: Only for hashtag-based
and query-based scraping.
usage:
python scraper -t 500 -ht=python --latest
--top : Twitter top tweets (default: False).
Note: Only for hashtag-based
and query-based scraping.
usage:
python scraper -t 500 -ht=python --top
```
### Sample Scraping Commands
- **Custom Limit Scraping**
```bash
python scraper -t 500
```
- **User Profile Scraping**
```bash
python scraper -t 100 -u elonmusk
```
- **Hashtag Scraping**
- Latest
```bash
python scraper -t 100 -ht python --latest
```
- Top
```bash
python scraper -t 100 -ht python --top
```
- **Query or Search Scraping**
_(Also works with twitter advanced search.)_
- Latest
```bash
python scraper -t 100 -q "Jak Roberto Anti Selos" --latest
```
- Top
```bash
python scraper -t 100 -q "International News" --top
```