Drill Hackathon (Sept 2017)

Drill Team at MapR

<P><STRONG>Event Agenda:</STRONG></P>
<P><A HREF="https://docs.google.com/spreadsheets/d/1PEpgmBNAaPcu9UhWmZ8yPYtXbUGqOAYwH87alWkpCic/edit?usp=sharing" TARGET="_blank" TITLE="Event Details" DATA-MSYS-CLICKTRACK="0" REL="nofollow noopener noreferrer">https://docs.google.com/spreadsheets/d/1PEpgmBNAaPcu9UhWmZ8yPYtXbUGqOAYwH87alWkpCic/edit?usp=sharing</A></P>
<P><STRONG>Meeting Room @ MapR Headquarters:</STRONG></P>
<P>Training Room East - Warriors; Training Room West - Sharks</P>
<P><STRONG></STRONG></P>
<P><STRONG>Target Audience: </STRONG></P>
<P><SPAN>This event is for anyone involved in the Apache Drill development – including contributors who are not committers or PMC members</SPAN></P>
<P><STRONG></STRONG></P>
<P><STRONG>WebEx for Remote Attendees:</STRONG></P>
<P><SPAN>JOIN WEBEX MEETING<BR><A HREF="https://mapr.webex.com/mapr/j.php?MTID=m9d39036e3953cce59ea81250c70c6c76" TARGET="_blank" DATA-MSYS-CLICKTRACK="0" REL="nofollow noopener noreferrer">https://mapr.webex.com/mapr/j.php?MTID=m9d39036e3953cce59ea81250c70c6c76</A><BR>Meeting number (access code): 806 111 950<BR>Meeting password: ApacheDrill<BR><BR></SPAN><STRONG></STRONG></P>
<P><STRONG></STRONG></P>
<P><STRONG>Event Details</STRONG></P>
<P>The goal is to get the community together for a day-long technical discussion on key topics in preparation for a Drill 2.0 release as well as potential improvements in upcoming 1.xx releases.  Depending on the interest areas, we could form groups and have a volunteer lead each group.</P>
<P>Based on prior discussions on the dev list, hangouts, and existing JIRAs, there is already a substantial set of topics and I have summarized a few of them below.   What other topics do folks want to talk about?   Feel free to respond to this thread and I will create a google doc to consolidate.</P>
<P>Understandably, the list would be long but we will use the hackathon to get a sense of a reasonable feature set for 1.xx and 2.0 releases.</P>
<DIV><BR></DIV>
<DIV>1. Metadata management</DIV>
<DIV>  1a: Defining an abstraction layer for various types of metadata: views, schema, statistics, security</DIV>
<DIV>  1b: Underlying storage for metadata: what are the options and their trade-offs?</DIV>
<DIV>      - Hive metastore</DIV>
<DIV>      - Parquet metadata cache (parquet specific)</DIV>
<DIV>      - An embedded DBMS</DIV>
<DIV>      - A distributed key-value store</DIV>
<DIV>      - Others..</DIV>
<DIV><BR></DIV>
<DIV>2. Drill integration with Apache Arrow</DIV>
<DIV>  2a: Evaluate the choices and tradeoffs</DIV>
<DIV><BR></DIV>
<DIV>3. Resource management</DIV>
<DIV>  3a: Memory limits per query</DIV>
<DIV>  3b: Spilling</DIV>
<DIV>  3c: Resource management with Drill on Yarn/Mesos/Kubernetes</DIV>
<DIV>  3d: Local vs. global resource management</DIV>
<DIV>  3e: Aligning with admission control/queueing</DIV>
<DIV><BR></DIV>
<DIV>4. TPC-DS coverage and related planner/operator enhancements</DIV>
<DIV>  4a: Additional set operations: INTERSECT, EXCEPT</DIV>
<DIV>  4b: GROUPING SETS, ROLLUP, CUBE support</DIV>
<DIV>  4c: Handling inequality joins and cartesian joins of non-scalar inputs (via Nested Loop Join)</DIV>
<DIV>  4d: Remaining gaps in correlated subquery</DIV>
<DIV>  4e: Statistics: Number of Distinct Values, Histograms</DIV>
<DIV><BR></DIV>
<DIV>5. Schema handling</DIV>
<DIV>  5a: Creation, management of schema</DIV>
<DIV>  5b: Handling schema changes in certain common cases</DIV>
<DIV>  5c: Schema-awareness</DIV>
<DIV>  5d: Others TBD</DIV>
<DIV><BR></DIV>
<DIV>6. Concurrency</DIV>
<DIV>  6a: What are the bottlenecks to achieving higher concurrency</DIV>
<DIV>  6b: Ideas to address these..e.g async execution?</DIV>
<DIV><BR></DIV>
<DIV>7. Storage plugins,  REST APIs related enhancements</DIV>
<DIV>    - Topics TBD</DIV>
<DIV><BR></DIV>
<DIV>8. Performance improvements</DIV>
<DIV>  8a: Filter pushdown</DIV>
<DIV>  8b: Vectorized Parquet reader</DIV>
<DIV>  8c: Code-gen improvements</DIV>
<DIV>  8d: Others TBD</DIV>

MapR Headquarters

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Drill Hackathon (Sept 2017)

More events from Drill Team at MapR

Discover more events from Drill Team at MapR, from Science & Tech to other experiences you might love.

Drill Hackathon (Sept 2017)

More events from Drill Team at MapR

Discover more events from Drill Team at MapR, from Science & Tech to other experiences you might love.

More Science & Tech events

Browse more Science & Tech events with different dates, prices, and formats to find your next great experience.