My script uses YouTube API V3 to pull information for each video within a playlist.
I store the playlists information in a file called Playlists.csv
I start by pulling the playlist entry from Playlists.csv and store it within the variable current_playlist whereby it is used to extract video information using the YouTube API model.
For Playlists with less than 50 videos, we need only carry out Step 1.
For Playlists with more than 50 videos, we carry out Step 2 until all videos are pulled (YouTube Restrictions).
Step 3 displays all videos captured for their respective playlists
ISSUE
I need a way to group all the scraped videos into one list, so that I can export them to an Excel file.
Currently only the last scraped entry is saved to Excel.
Playlists.csv
Playlist_id,Channel_name,qty PL2zq7klxX5AQU-VdQ2_EM4E3VY5KDs7M8,Ken Jee,3 PLEiEAq2VkUUKmhU6SO2P73pTdMZnHOsDB,Simplilearn,66 PLeo1K3hjS3uu4Lr8_kro2AqaO6CFYgKOl,codebasics,7CODE
import pandas as pdimport csvfrom urllib import request, responsefrom googleapiclient.discovery import build# INITIATE YOUTUBE APIapi_key = 'YouTube API Key goes here'youtube = build('youtube', 'v3', developerKey=api_key)# READ CSV FILEnumber_of_rows = pd.read_csv('Playlists.csv')print('\nNumber of Rows:\n', number_of_rows) # display datasetprint("Number of lines present:-", len(number_of_rows), '\n\n')# SCRAPE YOUTUBEdef run_scraper(): # Read CSV to List with open("Playlists.csv", "r") as f: csv_reader = csv.reader(f) next(csv_reader) for csv_line_entry in csv_reader: current_playlist = csv_line_entry[0].strip() # STEP 1: Scrape 1st 50 videos in Playlist as restricted by YouTube all_videos = [] request = youtube.playlistItems().list( part="snippet,contentDetails", maxResults=50, playlistId= current_playlist) response = request.execute() #print(response) for i in range(len(response['items'])): single_video = dict(Playlist_id = response['items'][i]['snippet']['playlistId'], Channel_id = response['items'][i]['snippet']['channelId'], Channel_name = response['items'][i]['snippet']['videoOwnerChannelTitle'], video_id = response['items'][i]['contentDetails']['videoId'], Video_published = response['items'][i]['snippet']['publishedAt'], Title = response['items'][i]['snippet']['title'], Description = response['items'][i]['snippet']['description'], Playlist_position = response['items'][i]['snippet']['position'], Videos_in_playlist = response['pageInfo']['totalResults']) all_videos.append(single_video) # STEP 2: Scrape upto an additional 50 further videos in playlists next_page_token = response.get('nextPageToken') more_pages = True while more_pages: if next_page_token is None: more_pages = False else: request = youtube.playlistItems().list( part="snippet,contentDetails", maxResults=50, playlistId= csv_line_entry[0].strip(), pageToken = next_page_token) response = request.execute() for i in range(len(response['items'])): single_video = dict(Playlist_id = response['items'][i]['snippet']['playlistId'], Channel_id = response['items'][i]['snippet']['channelId'], Channel_name = response['items'][i]['snippet']['channelTitle'], video_id = response['items'][i]['contentDetails']['videoId'], Video_published = response['items'][i]['snippet']['publishedAt'], Title = response['items'][i]['snippet']['title'], Description = response['items'][i]['snippet']['description'], Playlist_position = response['items'][i]['snippet']['position'], Videos_in_playlist = response['pageInfo']['totalResults']) all_videos.append(single_video) next_page_token = response.get('nextPageToken') # STEP 3: Display videos for all Playlists pd_all_videos = pd.DataFrame(all_videos) print('Videos In Playlist -', csv_line_entry[0],':\n', pd_all_videos) # Send Results to Excel pd_all_videos.to_excel('sample.xlsx', index=False) return run_scraper()OUTPUT
We can see from the image that all 3 playlists are pulled from the Playlists.csv file and all unique videos are scraped for their respective playlist.
I need a way to group all videos from all 3 playlists into one list. This should make the final list appear to have a quantity of 76 (3+66+7). Furthermore, we should be able to export all 76 video entries to Excel.
Any help would be much appreciated.
