"not Found: Table" For New Bigquery Table

I use the python sdk to create a new bigquery table: tableInfo = { 'tableReference':{ 'datasetId':datasetId, 'projectId':projectId,

Solution 1:

Per your answers to my question regarding using NOT_FOUND as an indicator to create the table, this is intended (though admittedly somewhat frustrating) behavior.

The streaming insertion path caches information about tables (and the authorization of a user to insert into the table). This is because of the intended high QPS nature of the API. We also cache certain negative responses in order to protect again buggy or abusive clients. One of those cached negative responses is the non-existence of a destination table. We've always done this on a per-machine basis, but recently added an additional centralized cache, such that all machines will see the negative cache result almost immediately after the first NOT_FOUND response is returned.

In general, we recommend that table creation not occur inline with insert requests, because in a system that is issuing thousands of QPS of inserts, a table miss could result in thousands of table creation operations which can be taxing on our system. Instead, if you know the possible set of tables beforehand, we recommend some periodic process that performs table creations in advance of their usage as a streaming destination. If your destination tables are more dynamic in nature, you may need to implement a delay after table creation has been performed.

Apologies for the difficulty. We do hope to address this issue, but we don't have any timeframe yet for doing so.

Solution 2:

Sean/Nils, even I have the same problem. I try streaming insert first and if this fails, I go ahead and create the table. But after table creation, streaming insert still failing for some more time(~ 30 secs to 4 mins) with error "Table not found". After this interval, same streaming inserts goes through without any issue.