104: Exploring Our Data with Chris Grainger

Chris Grainger explains Explorer, Nx's recent addition. It's a “Data Frame” library which sounds far more nebulous than it is. It lets us explore and manipulate two-dimensional data like we already do with DB tables, spreadsheets, CSV files, etc. Only this works in memory, is faster, and has predefined functions that make it very powerful. Explorer fills an important slot in the overall pie for data science and ML in Elixir. However, Explorer is also one of the tools that “regular” Elixir devs can take advantage of today!
Show Notes online - http://podcast.thinkingelixir.com/104 (http://podcast.thinkingelixir.com/104)
Elixir Community News
- https://twitter.com/josevalim/status/1535008937640181760 (https://twitter.com/josevalim/status/1535008937640181760) – At ElixirConf EU, José Valim announced there is an ongoing PhD scholarship for researching and developing a type system for Elixir that is powered by set-theoretic types.
- https://www.irif.fr/~gc/ (https://www.irif.fr/~gc/) – Giuseppe Castagna's Home Page. Senior Research Scientist
- https://arxiv.org/abs/2111.03354 (https://arxiv.org/abs/2111.03354) – Programming with union, intersection, and negation types
- https://twitter.com/fhunleth/status/1534826558774464512 (https://twitter.com/fhunleth/status/1534826558774464512)
- https://podcast.thinkingelixir.com/72 (https://podcast.thinkingelixir.com/72) – Typecheck interview
- https://github.com/ergo-services/ergo (https://github.com/ergo-services/ergo) – We learned of a Golang library called “ergo” that is an “actor based Framework for creating microservices using technologies and design patterns of Erlang/OTP in Golang”
- https://twitter.com/erlangforums/status/1534822378722713601 (https://twitter.com/erlangforums/status/1534822378722713601) – Patch Package OTP 25.0.1 Released
- https://erlangforums.com/t/patch-package-otp-25-0-1-released/1526 (https://erlangforums.com/t/patch-package-otp-25-0-1-released/1526) – Minor bug fixes. Zlib updated
- https://flowbite.com/docs/getting-started/introduction/ (https://flowbite.com/docs/getting-started/introduction/) – More free alternative to TailwindUI
- https://github.com/tailwindlabs/tailwindcss/releases/tag/v3.1.2 (https://github.com/tailwindlabs/tailwindcss/releases/tag/v3.1.2) – tailwindcss CLI v3.1.2 released which includes postcss-import into the CLI bundle
- https://twitter.com/adamwathan/status/1535989815778463746 (https://twitter.com/adamwathan/status/1535989815778463746) – Adam Wathan, the Tailwind CSS guy, had a fun quote that we liked.
- “Open source is “right to repair” for software — nothing more, nothing less. Maintain that attitude and any frustration or entitlement you feel will be replaced with gratitude. Better for maintainers, and better for you.”
Do you have some Elixir news to share? Tell us at @ThinkingElixir (https://twitter.com/ThinkingElixir) or email at [email protected] (mailto:[email protected])
Discussion Resources
- https://github.com/elixir-nx/explorer (https://github.com/elixir-nx/explorer)
- https://hexdocs.pm/explorer/Explorer.html (https://hexdocs.pm/explorer/Explorer.html)
- https://hexdocs.pm/explorer/Explorer.DataFrame.html (https://hexdocs.pm/explorer/Explorer.DataFrame.html)
- https://hexdocs.pm/explorer/Explorer.Series.html (https://hexdocs.pm/explorer/Explorer.Series.html)
- https://twitter.com/cigrainger/status/1518896955921154048 (https://twitter.com/cigrainger/status/1518896955921154048) – Tweet about using Rustler and Rustler Precompiled
- https://hexdocs.pm/explorer/exploring_explorer.html (https://hexdocs.pm/explorer/exploring_explorer.html) – Includes a “Ten Minutes to Explorer” Livebook guide in the docs to walk you through setup to getting started with a good overview.
- https://www.cigrainger.com/posts/introducing-explorer/ (https://www.cigrainger.com/posts/introducing-explorer/)
- https://databricks.com/glossary/what-are-dataframes (https://databricks.com/glossary/what-are-dataframes)
- https://pandas.pydata.org/docs/user_guide/dsintro.html (https://pandas.pydata.org/docs/user_guide/dsintro.html)
- https://dataframes.juliadata.org/stable/ (https://dataframes.juliadata.org/stable/)
- https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html (https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html)
- https://github.com/dashbitco/table (https://github.com/dashbitco/table)
- https://huggingface.co/ (https://huggingface.co/)
- https://www.amplified.ai/ (https://www.amplified.ai/) – Chris' company
- https://pandas.pydata.org/ (https://pandas.pydata.org/)
- https://en.wikipedia.org/wiki/Extract%2Ctransform%2Cload (https://en.wikipedia.org/wiki/Extract%2C_transform%2C_load)
- https://github.com/tyrchen/ex_polars (https://github.com/tyrchen/ex_polars) – Library that got him started with Rust, Rustler and working with polars
- https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html (https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html)
- https://github.com/philss/rustler_precompiled (https://github.com/philss/rustler_precompiled)
- https://huggingface.co/docs/transformers/main_classes/tokenizer (https://huggingface.co/docs/transformers/main_classes/tokenizer)
- https://github.com/elixir-nx/tokenizers (https://github.com/elixir-nx/tokenizers)
- https://docs.rs/crate/polars/0.22.0 (https://docs.rs/crate/polars/0.22.0)
- https://github.com/pola-rs/polars (https://github.com/pola-rs/polars)
- https://parquet.apache.org/ (https://parquet.apache.org/)
- https://avro.apache.org/ (https://avro.apache.org/)
- https://arrow.apache.org/overview/ (https://arrow.apache.org/overview/)
- https://vega.github.io/vega-lite/ (https://vega.github.io/vega-lite/)
- https://www.rdocumentation.org/packages/dplyr/ (https://www.rdocumentation.org/packages/dplyr/)
Guest Information
- https://twitter.com/cigrainger (https://twitter.com/cigrainger) – on Twitter
- https://github.com/cigrainger/ (https://github.com/cigrainger/) – on Github
- https://cigrainger.com (https://cigrainger.com) – Blog
Find us online
- Message the show - @ThinkingElixir (https://twitter.com/ThinkingElixir)
- Email the show - [email protected] (mailto:[email protected])
- Mark Ericksen - @brainlid (https://twitter.com/brainlid)
- David Bernheisel - @bernheisel (https://twitter.com/bernheisel)
- Cade Ward - @cadebward (https://twitter.com/cadebward)

Chris Grainger explains Explorer, Nx's recent addition. It's a “Data Frame” library which sounds far more nebulous than it is. It lets us explore and manipulate two-dimensional data like we already do with DB tables, spreadsheets, CSV files, etc. Only this works in memory, is faster, and has predefined functions that make it very powerful. Explorer fills an important slot in the overall pie for data science and ML in Elixir. However, Explorer is also one of the tools that “regular” Elixir devs can take advantage of today!

Show Notes online - http://podcast.thinkingelixir.com/104

Elixir Community News

https://twitter.com/josevalim/status/1535008937640181760 – At ElixirConf EU, José Valim announced there is an ongoing PhD scholarship for researching and developing a type system for Elixir that is powered by set-theoretic types.
https://www.irif.fr/~gc/ – Giuseppe Castagna's Home Page. Senior Research Scientist
https://arxiv.org/abs/2111.03354 – Programming with union, intersection, and negation types
https://twitter.com/fhunleth/status/1534826558774464512
https://podcast.thinkingelixir.com/72 – Typecheck interview
https://github.com/ergo-services/ergo – We learned of a Golang library called “ergo” that is an “actor based Framework for creating microservices using technologies and design patterns of Erlang/OTP in Golang”
https://twitter.com/erlangforums/status/1534822378722713601 – Patch Package OTP 25.0.1 Released
https://erlangforums.com/t/patch-package-otp-25-0-1-released/1526 – Minor bug fixes. Zlib updated
https://flowbite.com/docs/getting-started/introduction/ – More free alternative to TailwindUI
https://github.com/tailwindlabs/tailwindcss/releases/tag/v3.1.2 – tailwindcss CLI v3.1.2 released which includes postcss-import into the CLI bundle
https://twitter.com/adamwathan/status/1535989815778463746 – Adam Wathan, the Tailwind CSS guy, had a fun quote that we liked.
“Open source is “right to repair” for software — nothing more, nothing less. Maintain that attitude and any frustration or entitlement you feel will be replaced with gratitude. Better for maintainers, and better for you.”

Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected]

Discussion Resources

https://github.com/elixir-nx/explorer
https://hexdocs.pm/explorer/Explorer.html
https://hexdocs.pm/explorer/Explorer.DataFrame.html
https://hexdocs.pm/explorer/Explorer.Series.html
https://twitter.com/cigrainger/status/1518896955921154048 – Tweet about using Rustler and Rustler Precompiled
https://hexdocs.pm/explorer/exploring_explorer.html – Includes a “Ten Minutes to Explorer” Livebook guide in the docs to walk you through setup to getting started with a good overview.
https://www.cigrainger.com/posts/introducing-explorer/
https://databricks.com/glossary/what-are-dataframes
https://pandas.pydata.org/docs/user_guide/dsintro.html
https://dataframes.juliadata.org/stable/
https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html
https://github.com/dashbitco/table
https://huggingface.co/
https://www.amplified.ai/ – Chris' company
https://pandas.pydata.org/
https://en.wikipedia.org/wiki/Extract%2C_transform%2C_load
https://github.com/tyrchen/ex_polars – Library that got him started with Rust, Rustler and working with polars
https://hexdocs.pm/rustler_precompiled/RustlerPrecompiled.html
https://github.com/philss/rustler_precompiled
https://huggingface.co/docs/transformers/main_classes/tokenizer
https://github.com/elixir-nx/tokenizers
https://docs.rs/crate/polars/0.22.0
https://github.com/pola-rs/polars
https://parquet.apache.org/
https://avro.apache.org/
https://arrow.apache.org/overview/
https://vega.github.io/vega-lite/
https://www.rdocumentation.org/packages/dplyr/

Guest Information

https://twitter.com/cigrainger – on Twitter
https://github.com/cigrainger/ – on Github
https://cigrainger.com – Blog

Find us online

Message the show - @ThinkingElixir
Email the show - [email protected]
Mark Ericksen - @brainlid
David Bernheisel - @bernheisel
Cade Ward - @cadebward

Sponsored By: