Commander Bun Bun, landing the job!
DELETEand batch insert support for various JDBC-based connectors
The LinkedIn team covers the concept of the week in this section.
Today our PR of the week is from the future 🔮! LinkedIn is currently investigating the issue. This gives us a chance to talk about the research aspects that go into a PR.
With a view
V that performs a
UNION ALL from an old table
O and a new
datepartition values older than
D (say 2021-06-05),
O will be referred for data, while for date equal to or greater than
N will be used.
The query in question is:
SELECT * FROM V WHERE x IN (SELECT x2 FROM Z) AND cast(substring(datepartition,1,10) as date) >= date('2021-06-08')
Z has stats available and only have 17 rows in them. While the
data from view
V (which is entirely from underlying table
N for this query)
has say billions of rows.
This query used to take about 39 seconds to run before our upgrade (PrestoSQL-333). After the upgrade (Trino-352) it increased to approximately thirty-five minutes.
We actually covered the answer in episode 18.
You can use the Coral project that allows for translation between different SQL syntax. For example, it processes Hive QL statements and convert them to an internal representation using Apache Calcite. It then converts the internal representation to Trino SQL. See the docs for more details.
This diagram shows the creation of a Hive view, then shows the sequence of events when Trino reads that view.
Trino Meetup groups
If you want to learn more about Trino, check out the definitive guide from OReilly. You can download the free PDF or buy the book online.
Music for the show is from the Megaman 6 Game Play album by Krzysztof Słowikowski.