In our project we need to analyze comments including their replies based on custom criteria. Currently our solution is fetching all videos as well as consuming the CommentThreads endpoint provided by the YouTube Data API 3. All data will be fetched periodically, however to avoid exceeding the current quota limitations we're not able to fetch these data within a short period of time. The desired interval would be every 5 to 10 minutes. Depending on the traffic, it's very likely we'll exceed the quota limitations, especially if the connected channel has a vivid community with more than 1M followers posting thousands of comments a day.
To avoid any downtime in a production scenario, we would consider to request a quota increase. While filling out the form in the Google Cloud Console, we've noticed that exact figures are required.
The dilemma: At the moment we only can forecast the values, exact numbers we're currently not able to provide. This is only possible short before reaching the limits. On the other hand we can't risk any downtime and delaying the fetch intervals is also not an ideal approach.
Is it possible to apply for a quota increase based on forecasted calculations only? How would you handle connecting different customers/channels with huge traffic and the demand of short fetching intervals?