Welcome to the next edition of Vision Geek Newsletter, covering the essential news and learning resources in Computer Vision and Machine Learning, read by 1000+ Computer Vision practitioners.

Let’s get into it.

OpenAI Codex

OpenAI has introduced Codex, a powerful AI coding agent that converts natural language into code, streamlining programming tasks and enhancing developer productivity. It’s a cloud-based software engineering agent that can work on many tasks in parallel, powered by codex-1, a version of OpenAI o3 model optimized for software engineering.

It is not a full fledged IDE like Cursor. It is designed mainly to work with your GitHub repos. It can perform tasks like writing features, fixing bugs, answering codebase questions, and running tests (upto 30 minutes). Once it completes a coding task assigned, it can create a PR and a human can review it before merging it to the codebase providing more control to the human.

Codex-1 demonstrates high accuracy on internal (75%) and SWE (72.1%) benchmarks. The agents run in isolated containerized environments providing better security and autonomy. Codex is currently available to ChatGPT Pro, Team and Enterprise users today and will be available to Plus users soon.

OpenCV Conference

OpenCV recently hosted its first ever conference “OpenCV-SID Conference on Computer Vision & AI (OSCCA)” as part of Display Week 2025 (the premier international event for electronic display technologies) in partnership with SID (Society for Information Display).

It was an one-day in-person event featuring talks from industry experts such as Gary Bradski (Founder of OpenCV), Satya Mallick (CEO of OpenCV), Monica Song (Product Manager at Google AI Frameworks) and Joseph Nelson (CEO of Roboflow) etc. on various topics ranging from deep learning framework Keras to state-of-the-art augmented reality gaming to the latest advancements in single-shot detection.

Unfortunately, the recordings of the talks or slides are not publicly shared online. Hope to see relevant materials from the talks in other forms online in the near future.

Foundations of Computer Vision Book

“Foundations of Computer Vision” is a comprehensive textbook authored by Antonio Torralba, Phillip Isola, and William T. Freeman, published by the MIT Press in April 16,2024. Designed for students, educators, and practitioners, the book offers an in-depth exploration of computer vision, integrating both classical methods and contemporary deep learning advancements.

It incorporates the latest deep learning techniques, offering insights into how modern approaches have transformed computer vision. Beyond algorithms, the text addresses the relationship between machine vision and human perception, emphasizing the interdisciplinary nature of the field.

Its coverage includes transformers, diffusion models, statistical image models, fairness, ethics, and research methodologies, topics not commonly found in other textbooks. Concepts are presented in concise chapters with extensive illustrations, examples, and exercises to facilitate intuitive learning.

For those interested in a structured and modern exploration of computer vision, this textbook serves as a valuable resource, bridging traditional concepts with the latest advancements in the field. Having said that, it’s hard to cover everything in a single book. This book does not cover in depth the many applications of computer vision such as shape analysis, object tracking, person pose analysis and face recognition.

Fun Fact

It has taken 10 years to complete this book (started at Nov 2010) because of the rapid development in the field after the book started, mainly due to the explosion of deep learning.

Wildlife Species Detection Challenge

The Cup-ybara Challenge 2025 is an 8-week global AI competition hosted by Tryolabs (as part of its “AI for Good” initiative) on the Kaggle platform. It tasks participants with building machine-learning models to automatically identify native Uruguayan wildlife species in short camera-trap videos.

Tryolabs describes it as “an open competition to develop cutting-edge AI models for automated wildlife species detection”, aimed at speeding up a traditionally slow, manual monitoring process. The contest connects the data science community with conservation: the winning models will be shared with local NGO AMBÁ to be put into real-world use in Uruguay’s forests and reserves.

The dataset for this competition is composed of real-world footage collected by camera traps in Uruguayan ecosystems which includes unlabeled training data (train), labeled test data for public leaderboard evaluation (test), held-out private test data for final scoring (private_test).

Each video is exactly 15 seconds long, encoded in .mp4 format, and recorded using motion-triggered camera traps deployed in natural, uncontrolled environments.

• Competition start date: Monday, April 21st, 2025

• Submissions close date: Sunday, June 15th, 2025

During the competition, participants may collaborate in teams, iterate on models, and discuss ideas on the Kaggle forums. Kaggle enforces its standard competition rules: for example, no external data or manual re-labeling beyond what is provided, and a specific submission format (CSV) via the provided BaseModel class.

Submissions are evaluated by a “weighted F1 score” on the held-out test set, reflecting the multi-class classification accuracy of species labels. (Weighted F1 is common in Kaggle’s multi-category tasks)

The challenge is open to anyone worldwide. Participants need to register for free on Kaggle and accept the competition rules. Monetary prices will be awarded based on evaluation results performed after the main competition ends on a private dataset. (1st Place: $500, 2nd Place: $300, 3rd Place: $100, Honorable Mention: $100)

Prizes will be paid using Buy Me a Coffee. In summary, Cup-ybara aligns AI and wildlife conservation, inviting participants to leverage modern video classification techniques to help protect biodiversity.

Welcome to the next edition of Vision Geek Newsletter, covering the essential news and learning resources in Computer Vision and Machine Learning, read by 1000+ Computer Vision practitioners.

Let’s get into it.

LlamaCon 2025

Meta recently held it’s first-ever LlamaCon focusing entirely on Llama models, open source initiatives and the company’s work in AI.

Key Announcements & Highlights

• The conference kicked off with the showcase of the capabilities and strengths of the recently released natively multimodal Llama 4 models.

• A new standalone Meta AI app for users available on Android & iOS.

• Launch of new Llama API, hosted on their cloud which simplifies access to Llama models for inference in collaboration with Cerebras and Groq to deliver faster performance.

• The team also shared the technical details and behind the scene decisions while building Llama 4 models and the API.

• And showcased one-click API key generation, interactive playground for testing, SDKs for Python & TypeScript, compatibility with OpenAI SDKs, support for Llama 4 variants like Scout and Maverick and tools for fine-tuning the models with custom dataset.

• Meta has unveiled a suite of open-source Llama Protection tools and Llama Defenders Program aimed at improving AI safety and security.

• And announced 10 international recipients of the second Llama Impact Grants. With over $1.5 million USD awarded, these grants support companies, startups, and universities using Llama to drive transformative change.

• Announced that SAM 3 will be released this summer with support for text based prompts for segmenting objects/regions in images & videos whereas SAM 2 supported only clicks(points), masks and bounding boxes as prompts and showed a simple demo of SAM 3.

• Mark Zuckerberg’s a couple of fireside chats with Ali Ghodsi (CEO of Databricks) and Satya Nadella (CEO of Microsoft) and a Hackathon.

RF-DETR by Roboflow

Roboflow has recently released RF-DETR, real-time object detection transformer-based architecture under the Apache 2.0 license.

RF-DETR is the first real-time model to exceed 60 AP on the Microsoft COCO benchmark however models like DINO and SwinV2-G achieve slightly higher mAP scores on COCO, RF-DETR distinguishes itself by combining high accuracy with real time performance, making it particularly suitable for applications requiring both speed and precision. It also achieves state-of-the-art performance on RF100-VL, an object detection benchmark that measures model domain adaptability to real world problems.

The model comes in two variants: RF-DETR Base (29M parameters) and RF-DETR Large (129M parameters). RF-DETR is developed for projects that need a model that can run high speeds with a high degree of accuracy, and often on limited compute (like on the edge or low latency).

Key Benefits

• Converges faster than other models as pretraining helps.

• No NMS (smoother predictions between video frames and better understanding of overlapping objects)

• Gets the same results on cheaper hardware as LayerNorm is used instead of BatchNorm

• Multi Resolution training.

RT-DETR (Baidu) vs RF-DETR (Roboflow):

• Both RT-DETR and RF-DETR works on No NMS.

• In terms of speed, RT-DETR runs Up to 114 FPS whereas RF-DETR runs upto 25+ FPS.

• RF-DETR is designed for easier domain transfer & fine-tuning while RT-DETR transferability is not heavily optimized for small custom datasets.

RT-DETR is best suited when speed is of high priority than accuracy and RF-DETR is best suited when accuracy is of high priority while maintaining real time speed.

Cursor Free Plan

Cursor, the AI coding assistant built on top of VS Code, now offers a free student plan. You will have access to all Pro features for a year. This includes 500 fast premium requests per month and unlimited slow premium requests which usually costs $20 per month.

With this plan, students get access to key features like:

• AI pair programming (like GitHub Copilot)

• Smart code suggestions and autocompletion

• Inline code explanations and refactoring

• Help with debugging and navigating large codebases

Cursor is especially useful for beginners and those working on personal or academic projects in Python, JavaScript, TypeScript, and more. To get access, you’ll need to verify with a valid .edu email or upload proof of enrollment. Once approved, you’ll unlock premium features at no cost for a year.

However Cursor has restricted its free student plan in various regions including India. Cursor’s team says “we are working hard with our partners to bring this to more students soon”.

Students in the restricted regions can checkout other offers such as GitHub Copilot Student Pack, Microsoft Azure for Students in the meantime. Hopefully, as the program matures and verification workflows improve more regions will be added to the free-Pro list.

Welcome to the next edition of Vision Geek Newsletter, covering the essential news in Computer Vision and Machine Learning, read by 1000+ Computer Vision practitioners.

Let’s get into it.

Nvidia Jetson Orin Nano Super

Nvidia has announced a “super” update to their existing edge device “Jetson Orin Nano” and renaming the updated device to “Jetson Orin Nano Super” with a price drop from $499 to just $249 (8GB variant), claiming to be “The World’s Most Affordable Generative AI Computer”.

There is no hardware update. But through a pure software update, they have introduced a new 25W power mode to the device which has increased the performance in all aspects.

• 1.7x higher generative AI model performance.

• 67 Sparse TOPs, a significant increase from the previous 40 Sparse TOPs

• 102 GB/s of memory bandwidth, a significant leap from the previous 65 GB/s memory bandwidth.

• 1.7 GHz of CPU clock speed, up from 1.5 GHz.

• 1020 MHz of GPU clock speed, up from 635 MHz.

This compact yet powerful system can effortlessly handle a wide range of LLMs, VLMs, and Vision Transformers (ViTs), from smaller models to those with up to 8B parameters, such as the Llama-3.1-8B model.

For the given price point and the performance, Orin Nano Super can be a great choice as a baseline edge device to handle the modern day AI workloads. Though it is advertised as Generative AI computer, it can handle all forms of ML models.

PaliGemma 2

Google has released the next update to PaliGemma, open source vision-language model in the Gemma family of models. Like its predecessor, PaliGemma 2 uses the same powerful SigLIP and PaLI-3 for vision, but it upgrades to the latest Gemma 2 for the text decoder part.

What’s New

• Scalable performance: Optimize performance for any task with PaliGemma 2's multiple model sizes (3B, 10B, 28B parameters) and resolutions (224px, 448px, 896px), where as the previous version had only one variant (3B).

• Long captioning: PaliGemma 2 generates detailed, contextually relevant captions for images, going beyond simple object identification to describe actions, emotions, and the overall narrative of the scene.

• Expanding to new horizons: Leading performance on chemical formula recognition, music score recognition, spatial reasoning, and chest X-ray report generation, as detailed in the technical report.

PaliGemma 2 is distributed under the Gemma license, which allows for redistribution, commercial use, fine-tuning and creation of model derivatives. The pre-trained models have been designed for easy fine-tuning on custom datasets for specific tasks. Model files are available on Kaggle and HuggingFace. Get started with HuggingFace Transformers or Keras. (Docs | Notebooks | Demo)

Segment Anything Model (SAM) 2.1

Meta’s Fundamental AI Research (FAIR) team has released an update to SAM 2 featuring updated checkpoints with better accuracy.

What’s New

• Additional data augmentation techniques to simulate the presence of visually similar objects and small objects where SAM 2 previously struggled.

• Improved occlusion handling capability by training the model on longer sequences of frames and making some tweaks to positional encoding of spatial and object pointer memory (updated paper).

• SAM 2 Developer Suite, a package of open source code to make it easier than ever to build with SAM 2. This release includes training code for fine-tuning SAM 2 with your own data.

• Front-end and back-end code for the web demo.

Not much of changes in model sizes, no. of model variants and inference speed.

Meta has released the second version of SAM with significant improvements to the first version released last year. SAM2 can now segment and track selected objects in videos. This is made possible with the added memory bank in the architecture. SAM2 is 6x faster compared to SAM1 and the model weights sizes are much smaller. SAM2 has 4 model variants tiny, small, base plus and large. They have also released both the model weights and the dataset used for training. (github | blog | paper | demo)

FastHTML

Jeremy Howard and the team at answer.ai have released a new library for building modern interactive web apps with pure python in just few lines of code. FastHTML can be used for everything from collaborative games to multi-modal UI. It can be a potential alternative to existing libraries like streamlit, gradio and mesop. (docs | blog | github)

Google Imagen 3

Google announced it’s latest image generation model “Imagen 3” at Google I/O few months back. Now it is being released for people to use in select countries through ImageFX lab web interface and Google Cloud Vertex AI Studio. Imagen 3 seems to be capable of creating stunning photo realistic images from text prompts competing with other models like DALL-E 3, FLUX.1 and Midjourney with better safety guardrails.

Grok 2 with Vision

xAI, the AI lab from X (formerly Twitter) headed by Elon Musk, has released Grok 2 with vision capability. Grok started off as an LLM with just text input and output. The recent version expands it’s capabilities in vision and text understanding, also integrating real-time information from the 𝕏 platform. Currently it is in beta. 𝕏 Premium and Premium+ users will have access to two new models: Grok-2 and Grok-2 mini in the X app. xAI is working with Black Forest Labs to experiment with their uncensored image generation model FLUX.1

Mark’s Open Source Vision

Meta has been making significant contributions to the open source AI community lately with state-of-the-art foundational models like SAM2 and Llama 3.1 which competes with commercial models from companies like Google and OpenAI. If you have been wondering why Meta is actually open sourcing these models, Meta’s CEO Mark Zuckerberg shares his vision for open source in this Bloomberg interview.

If you are looking for some inspiring content to watch over the weekend in the AI space, below is a quick list of interesting videos for you.

Official PyTorch Documentary

PyTorch team has produced a documentary covering the journey of the library from the initial days and it’s impact on the ongoing AI revolution along with sponsors like AMD, AWS, Meta and Microsoft. The documentary unveils the authentic narrative of PyTorch’s inception, attributing its existence to a dedicated group of unsung heroes and shares the strength of the PyTorch community. Inspiring watch.

Andrej Karpathy’s Keynote Speech

Andrej Karpathy recently gave the keynote speech at the UC Berkeley AI Hackathon 2024 Awards ceremony. Apparently he loves the vibe of hackathons. He discussed about the current state of AI, his early days at OpenAI, LLM OS, his vision for AGI and how working on side projects have the potential to snowball into something bigger. This video also showcases the winning projects from the hackathon.

Open Source Alternative to GPT-4o

A French AI Lab called Kyutai Labs has released a real time voice enabled model called Moshi, a potential open source alternative to OpenAI’s GPT-4o. The team claims that in just 6 months, with a team of 8, the Kyutai research lab has developed the model from scratch and their mission is to build and democratize artificial general intelligence through open science (the real OpenAI?).

During the presentation, the Kyutai team interacted with Moshi to illustrate its potential as a coach or companion for example, and its creativity through the incarnation of characters in roleplays. Code and model weights to be open sourced soon. Chat with Moshi at moshi.chat. Currently conversations are limited to 5 mins.

Andrew Ng on AI Agentic Workflows

Andrew Ng recently gave a talk at Sequioa’s AI Ascent event on AI agentic workflows. “AI Agents” has become a buzzword recently and being tossed around in many places. In this talk, Andrew Ng presents concrete examples of agentic workflow and how it can improve the performance of LLMs. He also mentioned his experience using an open source version of Devin (the AI Software Engineer) - ChatDev.

Hope you find these videos engaging. Let us know what you have been watching lately.

Cheers.

Tesla Bot

Last year Elon Musk announced that Tesla will be building a humanoid robot in their AI Day event and revealed the conceptual outlook of the Tesla Bot. “Tesla AI Day 2022” was held recently and the team showcased the prototype they have been working on. There are two variants Bumblebee and Optimus.

The team started off with Bumblebee where they used readily available off-the-shelf hardware components and moved on to Optimus where all the hardware components are designed by Tesla in-house, giving them a lot more control and performance.

Most of the software built for Autopilot in the cars is repurposed for the Bot. We have seen impressive humanoid robots from other companies in the past, they are very expensive and made in small quantities.

What makes Tesla Bot interesting is that they are planning to build these Bots in large quantities such that the cost of a bot will become lower than a car, hopefully less than $20,000. They already have the hardware, AI software and real world data from the cars. Tesla has the potential to make humanoid robots mainstream. We just have to wait and see if they can pull this off.

Autopilot

The Autopilot team gave a glimpse of how they are handling complex real world scenarios. They are slowly rolling out FSD (Full Self Driving) Beta to more and more customers.

One interesting thing about Tesla is that they don't use LiDAR at all in their cars. Everything is done using the 8 cameras present in the vehicle. It's quite hard. Most of the self driving car companies use expensive LiDARs.

They showcased the Occupancy Network they use to predict what's present on the scene around the vehicle in 3D and how they are taking inspirations from language modelling to tackle lane prediction at complex intersections.

They have developed their own compiler, file format (.smol) and neural network accelerator (TRIP Engine) to make the best use of their FSD hardware. They also use simulation to create 3D scenes to create training data of specific scenarios to improve the model accuracy, completely automated, no 3D artists involved. It's fascinating to see how the team is optimizing every single aspect to achieve better performance and accuracy.

Dojo Supercomputer

Tesla gets tens of thousands of real world video clips from the cameras on their fleet everyday. Training machine learning models on this ever-growing dataset is not an easy task.

Tesla uses a cluster of 14,000 GPUs. Out of which 10,000 GPUs are used for model training and 4000 GPUs are used for auto labeling task. Even with this massive GPU cluster, it takes months to train the models on huge datasets.

The team has built a supercomputer named "Dojo" to tackle this problem. Designing the entire hardware and software stack from the ground up has given them a huge boost in performance at a fraction of the usual GPU cost. The same models can now be trained in less than a week instead of a month.

From building a completely new compiler to creating a customer protocol (TTP - Tesla Transfer Protocol) to building a custom network interface card (DNIC - Dojo Network Interface Card) the team has innovated at all the layers. Truly inspiring work.

Meta Universal Speech Translator

Eight months ego Meta announced their “Universal Speech Translator” project, which aims to develop new AI methods that will allow real-time speech-to-speech translation across many languages. Now the team has come up with an interesting demo.

So far, AI powered speech translation systems convert the speech to text first and use NLP (Natural Language Processing) models to understand the text and convert it to text in the destination language and then convert that text to speech.

This approach works well for the languages that have a standard writing system. But nearly half of the world’s 7,000+ living languages are primarily oral and do not have a standard or widely used writing system.

This makes it impossible to build machine translation tools using standard techniques, which require large amounts of written text in order to train language models.

To address this challenge, Meta’s AI team has built the first AI-powered speech-to-speech translation system for Hokkien, a primarily oral language that’s widely spoken in China, Taiwan and few other countries but lacks a standard written form.

The team has developed a variety of novel approaches and systems to achieve this. Meta is open-sourcing the translation models, evaluation datasets and research papers so that others can reproduce and build on their work.

Though the Hokkien translation model is still a work in progress and can translate only one full sentence at a time, it has the potential to allow anyone to communicate with anyone else from anywhere in the world in their own native language. Exciting times we live in !

Google AI@22

In the recent Google event “AI @ 22” , their research team shared the recent advancements in generative modelling. There has been a lot of improvement particularly in image and video generation from text.

“Imagen” and “Parti” are two models the team has built with slightly different approaches to generate images from text. The results from these models are so crisp and of very high quality. We have definitely come a long way from generating low resolution handwritten numbers with MNIST.

Now that we are able to generate realistic high definition images from text, naturally the next step is to try generating videos from text. “Imagen Video” and “Phenaki” are two such attempts in creating consistent videos from text.

Taking inspirations from language modelling in solving computer vision problems has become common now. “Parti” and “Phenaki” are two such examples. Text-to-video models are still in its infancy but the revolution has definitely started.

NN Zero-to-hero

In case you noticed, in the Tesla AI Day 2022 event, Autopilot session was not presented by Andrej Karpathy. He was leading the AI team at Tesla until last year.

It seems that after a four month break from work, Andrej has decided to leave Tesla. For his tweet, Elon Musk has replied as "Thanks for everything you have done for Tesla! It has been an honor working with you.“

Andrej has mentioned that he has no concrete plans for what’s next but look to spend more time revisiting his long-term passions around technical work in AI, open source and education. He is currently working on creating a course called “Neural Networks - Zero to Hero”. He is making the course available for free to everyone.

It is hosted on GitHub. Course lectures are available as YouTube videos and the coding exercises are available as Google colab notebooks. It is currently in progress. Some of the videos and notebooks are already available.

PyImageSearch got acquired

PyImageSearch is one of the widely read blogs in the field of Computer Vision and Deep Learning. If you are a computer vision practitioner, most probably you might have read some of the blog posts on PyImageSearch in your learning journey.

Adrian Rosebrock is the core author and owner of the blog. It seems that PyImageSearch has been acquired last year. The details about the acquisition are not yet publicly disclosed.

In his new venture Info Product Mastery, Adrian has mentioned that

“I launched my company, PyImageSearch.com, in 2014. By 2017 I had grown it to 7 figures per year in revenue by selling eBooks and online courses I had created. In 2021 PyImageSearch was acquired for a life changing exit. I’m here to share my experiences so you can learn from the mistakes I’ve made while building and growing your own info product business.”

The podcast aims to help developers, educators, and entrepreneurs launch and grow their online education businesses. Whether they are just looking to create a passive income stream, or build a full-time living.

AI fun :)

Support this newsletter ❤️

If you are getting value out of my work, consider supporting me on Patreon and unlock exclusive benefits.

Become a Patron

deeplearning.ai has launched a new specialization on Coursera containing 3 courses on Generative Adversarial Networks (GAN). Since it’s inception in 2014 by Ian Goodfellow, GANs have created a whole new subfield in AI, giving machines the ability to imagine and learn to create new content (text, image, video, music etc). This is already proving to be a game changer (good or bad ?) with things like OpenAI GPT-3 and DeepFakes. Apart from this book (uses TensorFlow/Keras), this set of courses are going to be really helpful for people interested in getting a better understanding of GANs and get hands-on with PyTorch.

Nvidia GTC Fall 2020

Nvidia has announced a bunch of new products and partnerships in the GPU Technology Conference (GTC) happened virtually in October. Ranging from a new 2GB Jetson Nano ($59) to data center specific products like DPU, new architectures like Bluefield in partnership with ARM and VMWare. Nvidia is definitely spearheading the AI hardware movement in full throttle. They have completely repositioned themselves from just a gaming company to premier AI computing company.

Transformers for Image Recognition

CNNs have been the go-to algorithm/method for computer vision tasks in almost the last decade since AlexNet won the ImageNet competition in 2012. In recent times, Transformer networks have been slowly gaining momentum since the success of models like GPT-3. Though transformer networks have been mainly used in NLP, it is slowly getting into computer vision as well.

Facebook AI Research team successfully applied transformers to object detection recently, OpenAI team also released Image-GPT. And now a team from Google has used transformers for image classification. The paper is currently under review. Can transformers replace CNN ? well, we have to wait and see. But CNN is definitely starting to face competition. (paper | github)

Deep Learning with PyTorch Book

Just in case you missed it, few months back PyTorch team has made the full version of the book “Deep Learning with PyTorch” freely available for everyone. Apart from the official PyTorch tutorials, this book will be a great place to start for anyone new to PyTorch. You can download the book here.

OpenCV 4.5.0

OpenCV 4.5.0 has been released. Some of the improvements include

• Better SIFT in the main repository

• Real-time Single Object Tracking using Deep Learning

• (opencv_contrib): OpenCV bindings for Julia Programming Language

• (dnn module) 3-5x faster inference on ARM, Improved ONNX support, fixes and optimizations in DNN CUDA backend.

• and the license has been switched to Apache 2 from BSD 3-clause license. It doesn’t affect how we use OpenCV. It’s still free to use in all forms (including commercial use), Apache 2 license provides more legal protections. Read more here for the rationale behind this decision.

AI Pay Grades

Though there is a high demand for good AI talents in the industry, it may be unclear what the actual job offers look like around the world. Often times, ML Engineers like us are not really sure whether they are underpaid and (ironically) don’t have enough data to get an overall idea and better negotiate the compensation. aipaygrad.es is a good initiative from the community to collect and show statistics from real job offers made.

Keep in mind, currently there is only a small amount of data available and mostly from US. So it’s not yet fully representative of the industry. Nevertheless keep an eye on this site for more data over a period and if possible spread the word and contribute.

Landing an ML job

Even though many studies say that there is a shortage of ML Engineers, getting into an actual ML role might not be that easy. In this session hosted by deeplearning.ai , technical recruiters from top companies like Pinterest and Scale AI explain practical approaches on how to land an ML job and build a career in the AI industry. Informative session for ML Engineers.

AI fun :)

Support this newsletter ❤️

If you are getting value out of my work, consider supporting me on Patreon and unlock exclusive benefits.

Become a Patron

Transformer networks have gained a lot of popularity in the recent years. But mostly in the sequence modeling space like NLP, not much used in the computer vision space.

Recently Facebook AI research team has successfully used transformer networks for object detection making the end-to-end detection pipeline much simpler. (paper | code | blog)

Nvidia Jetson Xavier NX DevKit

Nvidia announced Jetson Xavier Nx last November but it wasn’t available to buy right away. After the wait, the devkit has finally arrived.

What’s new ?

Nvidia announced Jetson Nano last year as entry level edge device for AI at $99. Nano was really well received in the market. Now Nvidia is upping the game with Xavier NX for heavy compute applications on edge which Nano can’t handle. In fact, Nvidia calls it “the World’s Smallest AI Supercomputer”. It comes at the same form factor as Nano but with way better compute capability at $399.

Google opensources Big Transfer (BiT)

Researchers at Google have found that pre-training computer vision models on very large scale datasets (way bigger than ImageNet) helps the model to learn quickly and generalize better when finetuned using transfer learning for other tasks even with vey small training data. This approach has been found useful in the language domain in recent times. This work in computer vision shows the need for very large scale open datasets (one of the datasets with 300M images used in the research was an internal dataset). (paper | code | blog)

AI for Healthcare Nanodegree

Following Coursera, Udacity has also launched “AI for Healthcare” nanodegree, a specialized course focusing on applying machine learning to medical data. With everything going on currently, applying AI to healthcare has gained more importance than ever. Checkout the curriculum to learn more.

CVPR 2020

CVPR this year was conducted as a virtual conference due to the pandemic. Featured two special keynotes from Satya Nadella, CEO of Microsoft and Charlie Bell, SVP of AWS. Many tutorial and workshop organizers have generously posted the video recordings online. Checkout the individual websites for the tutorial / workshops you are interested in.

Deep learning lectures from DeepMind

DeepMind in collaboration with UCL has shared a series of deep learning lectures by research scientists at deepmind. 10 out of 12 planned lectures are available on YouTube. Probably the course is affected due to the pandemic. Nevertheless, the available lectures are definitely worth checking out.

Fei-Fei Li on Exponential View Podcast

Dr.Fei-Fei Li, creator of ImageNet was featured on Exponential View podcast. She discussed the early days of ImageNet, the impact it has created, her current work on healthcare and the vision of Human centered AI Lab (HAI) she has cofounded at Stanford. Checkout the full episode here or wherever you generally consume podcasts.

AI fun :)

You might have heard by now, YOLOv4 has been released. But not actually from the original author of YOLO, Joseph Redmon. This is from different researchers including Alexey Bochkovskiy who is known for his popular github repo on DarkNet, forked from original DarkNet repo. He has made several improvements to DarkNet. In a way his repo was more popular than the original one.

YOLOv4 employs more modern state-of-the-art techniques such as Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), Cross mini-Batch Normalization (CmBN), Self-adversarial-training (SAT) and Mish-activation to achieve better speed and accuracy. Checkout the paper and code for more details.

Nvidia DGX A100

Nvidia CEO Jenson Huang announced new products in his “kitchen keynote” as part of GTC 2020. The highlight was the absolute GPU beast DGX A100 which is aimed at datacenter usage or as a server node for high intensive training/inference for your research team, amazingly priced at $199K (yes, you read that right) along with other announcements real time ray tracing and audio-to-face (A2F).

PyTorch 1.5

PyTorch 1.5 has been released with several changes and additions along with updates for torchvision, torchaudio and torchtext. Facebook (maintainer of PyTorch) is also partnering with AWS for TorchServe, a tool that helps you deploy your PyTorch models to the cloud and use them in production with API calls over the internet. Checkout the official blog post for more details.

Machine Learning Yearning

Doing online courses and assignments is one thing, applying machine learning in the real world and deploying the model to production is completely a different thing. Dr. Andrew Ng shares his practical experience building real world products at companies like Baidu and Google.

This book is a treasure trove of practical knowledge and best practices for AI Engineers. Be sure to get the draft of the book for free here. A must read.

Tesla at ScaledML Conference

Andrej Karpathy (Director of AI at Tesla) gave a talk at ScaledML conference 2020 on how the company is using AI to get closer to Full Self Driving. Particularly he discussed about stop sign detection which looks like a simple problem but in reality how challenging it is in production. Also he discussed how Tesla is achieving these results without actually having a LiDAR, just with cameras and few other sensors (radar, ultrasonic) which is quite difficult and impressive.

ICLR 2020

International Conference on Learning Representations (ICLR) is one of the premier conferences in the field of ML/AI. This year it was planned to be conducted at Ethiopia. But due to the current corona virus situation, the organizing committee has decided to make it a virtual conference going completely online.

I think this is the first premier conference to go completely online. Last year NeurIPS was live streamed online. But it was additional to the physical event happened in Canada. A portion of the talks (including the talks from Yann LeCun, Yoshua Bengio, Andrew Ng etc.) and workshops from ICLR have been made available online. Feel free to check it out.

The age of AI

YouTube has released an original series on AI featuring Robert Downey Jr. as host. The series explores the current state of the art works in AI, the impact it can have in our lives and what the future has in store for us. Interesting watch if you have some time to kill.

AI fun :)

Prof. Andrew Ng's venture landing.ai has developed a tool for monitoring social distancing. They have also shared the techniques on how they built it so that other people can build themselves a similar tool if needed (the actual source code is not open sourced though).

TensorFlow Dev Summit 2020

TensorFlow Dev Summit this year was a completely online event due to covid-19 concerns. All the talks are available on TensorFlow YouTube channel. In case you missed it, feel free to catch up.

AI for Medicine Specialization

deeplearning.ai has launched a new specialization on Coursera “AI for Medicine” consisting of three courses taught by experts from Stanford. If you are interested in applying AI in the healthcare space it's definitely worth checking out.

Covid-19 - a realistic look

Jeremy and Rachel from fast.ai has put together a realistic analysis on the current situation. As they point out, just trying to stay calm and not panic is not enough. Staying informed and preparing ourselves both mentally and physically is as important as staying sane in this difficult time.

GANs in Action

If you have been thinking to learn to work with GANs but didn’t really get hold of a good comprehensive resource then this book “GANs in Action - Deep learning with Generative Adversarial Networks” is for you. This manning publication is definitely one of the good resources out there on GAN covering from the basic idea to state of the art results.

TensorFlow without a PhD

If you are a fan of Martin Görner’s without a phd series, then you should definitely star this github repo. This repo contains the collection of resources for all the talks he has given in this “without a phd” series.

AlphaGO movie

DeepMind has released the full documentary “AlphaGO” on YouTube. Previously it was available on Netflix. Just to give some background, AlphaGO is the computer program that beat the 18 times World Champion Lee Sedol (Professional GO player from South Korea) in the board game “GO”.

GO is considered to be a complex game to solve for computer programs. AlphaGO beating a world champion is considered a major breakthrough in the history of AI. If you are looking for something inspiring to watch during this quarantine period, GO for it.

AI fun :)

Bonus:

How China tracks everyone

Sneak peak into how China does surveillance at scale.

TensorFlow team has teamed up with O’Reilly to host their first TensorFlow World conference earlier this month. If you are wondering how does this differ from TensorFlow Dev Summit, well, the key difference is in Dev Summit mostly people from the TensorFlow team will present their work. But TensorFlow World is a place for everyone in the community to learn and share what they are building with TensorFlow.

That means you can see talks and sessions from diverse set of people including TensorFlow team. All the sessions from TensorFlow team is up on TensorFlow YouTube channel. Other talks are on the O’Reilly online learning platform. O’Reilly says all the recorded sessions will be available on the platform after three weeks from the conference. They have a 10 day free trial. No credit card required. Give it a try to watch all the sessions.

Azure Kinect

Microsoft has released a new version of Kinect called ‘Azure Kinect DK’. DK stands for developer kit. Original version of Kinect was released almost a decade ago for Xbox. It was mainly intended for gaming use with Xbox. But people also used it for computer vision research because of the depth sensing capability it had.

This time Azure Kinect is solely intended for developers and companies to build things and not intended for regular consumers and this is not meant to replace the existing kinect for Xbox. Microsoft says they have put together their best sensors to build AI applications. It has a 12 MP RGB camera, 1 MP depth sensing camera and microphone arrays. It doesn’t have onboard processor but it can be connected to a CPU to process the wealth of information it captures to build vision and speech applications.

Real time video gesture recognition

Researchers at MIT have developed a new technique “Temporal Shift Module (TSM)” to do video classification efficiently on low compute devices. Generally doing video activity recognition in real time on edge devices is hard because of the high compute. In video classification we look at sequence of frames to predict the class as opposed to looking at a single frame at a time for image classification or object detection. The demo runs in real time on Nvidia Jetson Nano under 10 Watts. (Paper | GitHub | Site)

CVPR 2019

Computer Vision and Pattern Recognition (CVPR) is one of the premier conferences in computer vision. CVPR 2019 was over earlier this year. Not all of us can afford to travel and attend the conference in person. Luckily there is this thing called ‘internet’. Computer Vision foundation has made lot of the sessions available online. You can find the video recordings (if available) on YouTube and slides under each session page on the conference website. This really helps to get a sense of what’s going on in the research frontier.

Ancient Secrets of Computer Vision

Joseph Redmon (the author of YOLO/DarkNet) teaches a computer vision course at the University of Washington. He has generously posted the video lectures on YouTube. It’s definitely one of the good introduction to CV courses available online. Feel free to check it out.

Google Coral TPU graduates out of beta

Google launched it’s new hardware Coral Edge TPU earlier this year in March for AI at the edge. After six months now its stable and out of beta. It runs models in a specific TensorFlow Lite edgetpu format very efficiently for low latency real time applications. AI on the edge is off to a good start. Long way to go though !

3Blue1Brown on Siraj Raval Podcast

You might know Grant Sanderson from his awesome YouTube channel “3Blue1Brown”. He was recently interviewed by Siraj on his podcast where they discussed about doing math animations, Grant’s recent visit to India and more. Listen to the episode to learn more. It is available on Google Podcasts, Spotify and possibly wherever you consume your podcasts.

AI fun :)

University of Alberta and Alberta Machine Intelligence Institute (AMII) have come together to launch a specialization (4 courses) on Reinforcement Learning on Coursera. If you have been trying to learn RL, you might already know that there are not a lot of well structured go-to courses out there. There are some really good resources like Sutton & Barto’s text book, David Silver’s course lectures and UC Berkeley RL course lectures on YouTube, OpenAI’s spinning up in deep RL. But they lack well structured assignments or proper beginner-friendly MOOC setup.

Why this matters:

The reason this course looks promising is, it’s from the people who work directly with the great minds in RL like Prof. Sutton. In fact Sutton himself is involved in the creation of this course. I kind of feel like these courses might be the video version of Sutton’s text book. But it’s definitely worth checking out. Give it a try and let me know your thoughts if you manage to finish any of the courses in the specialization.

OpenAI at TechCrunch Disrupt SF

Sam Altman (CEO) and Greg Brockman (CTO) from OpenAI were at TechCrunch Disrupt San Francisco and discussed some of the company’s earlier decisions (forming capped for profit OpenAI LP, partnership with Microsoft, GPT-2 etc) and future roadmap for OpenAI. Greg also showed a demo of their recent experiment with multi-agent RL and how the agents discovered tool usage.

Tesla acquires DeepScale

Tesla has acquired DeepScale (the company behind SqueezeNet paper). SqueezeNet was one of the first attempts in creating smaller models without losing too much accuracy. This acquisition clearly shows the need for efficient models that can run on edge devices with smaller footprint. AI on edge is definitely booming.

Introduction to TensorFlow Lite on Udacity

TensorFlow Lite team has launched a course on Udacity covering deploying TFLite models to mobile and edge devices. Google also released their Coral Edge TPU earlier this year to advance AI applications on the edge. The course may not be advanced in-depth course. But it can definitely serve as a good comprehensive introduction to TensorFlow Lite.

PyTorch Mobile

Until now, if you want to deploy a model to mobile, your best bet was TensorFlow Lite. PyTorch has added support for mobile (Android and iOS) with their 1.3 release. (PyTorch is upping it’s game!)

DeepMind Podcast

DeepMind has completely restructured the organization recently. They have released a limited series podcast with mathematician Hanna Fry hosting the show and giving us an insider look. All the eight episodes are out. You can find it on Spotify, Google Podcasts or wherever you consume your podcasts.

PyTorch official YouTube channel

PyTorch gets it official YouTube Channel (finally!). I have been waiting for this. Earlier the videos were scattered among Facebook Developers YouTube channel and facebook pages. Now we can watch all the content in one place. Go ahead and watch the videos (PyTorch Developer Conference, Summer Hackathon etc) when you are free.

AI fun :)

Welcome to Vision Week - a biweekly (once in two weeks) newsletter on Computer Vision and AI.

It will be a healthy mix of carefully curated industry news, interesting articles/blog posts/interviews/podcasts, recent advancements, research papers, demos, learning resources and hidden gems in the field of computer vision.

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!