Posted on

Deep Dive Into Join Execution in Apache Spark

Join operations are often used in a typical data analytics flow in order to correlate two data sets. Apache Spark, being a unified analytics engine, has also provided a solid foundation to execute a wide variety of Join scenarios.

At a very high level, Join operates on two input data sets and the operation works by matching each of the data records belonging to one of the input data sets with every other data record belonging to another input data set. On finding a match or a non-match (as per a given condition), the Join operation could either output an individual record, being matched, from either of the two data sets or a Joined record. The joined record basically represents the combination of individual records, being matched, from both the data sets.

Posted on

7 Essential Tools for a Competent Data Scientist

A data scientist extracts manipulate and generate insights from humongous data. To leverage the power of data science, data scientists apply statistics, programming languages, data visualization, databases, etc.

So, when we observe the required skills for a data scientist in any job description, we understand that data science is mainly associated with Python, SQL, and R. The common skills and knowledge expected from a data scientist in the data science industry includes - Probability, Statistics, Calculus, Algebra, Programming, data visualization, machine learning, deep learning, and cloud computing. Also, they expect non-technical skills like business acumen, communication, and intellectual curiosity.

Posted on

3 Ways to Select Features Using Machine Learning Algorithms in Python

Artificial intelligence which gives machines the ability to think and behave like humans are gaining traction since the last decade. These features of artificial intelligence are only there because of its ability to predict certain things accurately, these predictions are based upon one certain technology which we know as machine learning (ML). Machine learning as the name suggests is the computer’s ability to learn new things and improve its functionality over time. The main focus of machine learning is on the development of computer programs that are capable of accessing data and using it to learn for themselves. 

To implement machine learning algorithms, two programming languages, R and Python for machine learning are normally used. Generally, selecting features for training data on machine learning in python is a very complex and technical process. But here we will go over some basic techniques and details regarding what is machine learning and how it works. So, let us start by going into detail regarding what ML is, what feature selection is and how can one select feature using python.

Posted on

8 Best Big Data Tools in 2020

In today’s reality, data gathered by a company is a fundamental source of information for any business. Unfortunately, it is not that easy to drive valuable insights from it.

Problems with which all data scientists are dealing are the amount of data and its structure. Data has no value unless we process it. To do so, we need big data software that will help us in transforming and analyzing data.

Posted on

Introducing Streamlit Components

In the ten months since Streamlit was released, the community has created over 250,000 apps for everything from analyzing soccer games to measuring organoids and from COVID-19 tracking to zero-shot topic classification. Inspired by your creativity, we added file uploaders, color pickers, date ranges, and other features. But as the complexity of serving the community grew, we realized that we needed a more scalable way to grow Streamlit’s functionality. So we’re turning to Streamlit’s best source of ideas: you!

Today, we are excited to announce Streamlit Components, the culmination of a multi-month project to enable the Streamlit community to create and share bits of functionality. Starting in Streamlit version 0.63, you can tap into the component ecosystems of React, Vue, and other frameworks. Create new widgets with custom styles and behaviors or add new visualizations and chart types. The possibilities are endless!

Posted on

Data Scientists Career Path: From Associate to Director Levels

It’s a data world now. Given the astronomical rise of data being churned out every day, businesses everywhere are interested to glean insights from their extensive data. They are becoming more reliant on data scientists to create business value from data collected.

This has led to an immediate spike in demand for data scientists. A data scientist is a relatively new career trajectory, where organizations hire them at various levels as junior, mid-level, senior, principal data scientist, and director.

Posted on

Predicting Wine Quality With Several Classification Techniques

Introduction

As the quarantine continues, I’ve picked up a number of hobbies and interests…including WINE. Recently, I’ve acquired a taste for wines, although I don’t really know what makes a good wine. Therefore, I decided to apply some machine learning models to figure out what makes a good quality wine!

Posted on

What Devs Are Working on During Covid

While we're still adapting to the onset of Covid-19, the world's still turning. And fortunately, developers are used to working in just about any environment and while overcoming plenty of diversity. 

About a month ago, we put out a call for interesting coronavirus stories — and you did not disappoint.

Posted on

6 Top Big Data and Data Science Courses to Start Learning Right Now

Among the most anticipated technology trends for the future, Big Data finds its place and offers an excellent opportunity to shape your career.

Big Data analytics has secured its place in the top technology trends for the year 2020. The increasing demand for AI and machine learning-enabled solutions drives the requirement for data scientists, and big data helps you pave your way to enjoying a successful career in the same.

Posted on

Book Review: Machine Learning With Python for Everyone by Mark E. Fenner

Machine learning, one of the hottest tech topics of today, is being used more and more. Sometimes it's the best tool for the job, other times a buzzword that is mainly used as a way to make a product look cooler. However, without knowing what ML is and how it works behind the scenes, it’s very easy to get lost. But this book does a great job of guiding you all the way from very simple math concepts to some sophisticated machine learning techniques. 

Today, in the Python ecosystem, we have a plethora of powerful data science and machine learning related packages available, like Numpy, Pandas, Scikit-learn, and many others, which help to simplify a lot of its inherent complexity. In case you are wondering, in terms of Python packages, the great hero in this book is Scikit-learn, often abbreviated as  sklearn. Of course, the data wrangling is much easier and much faster using Numpy and Pandas, so these two packages are always covering sklearn’s back. Seaborn and Matplotlib, two of the most standard data visualization packages for Python, are also used here. In chapter 10, patsy makes a brief appearance, and in chapter 15, pymc3 is used in the context of probabilistic graphic models. 

Posted on

Quick and Easy Configuration of Oracle Data Science Cloud Service

Hello to everyone,

To use some features of Oracle Data Science Cloud Service (to save models, to read basic data about OCI, to establish ADW or Object Storage connections), you need to configure this service when you first turn it on. A description of this configuration is described in the getting-started.ipynb notebook that comes in the service. I prepared a .sh considering that some steps can be automated in this recipe. Through this article, I will explain how to use this .sh quickly.

Posted on

Oracle Data Science Cloud Service

Oracle launched Data Science Cloud Service recently. This service, which can be used over the cloud, is actually a virtual machine and contains many pieces of open source software. With this service, it is aimed to develop a ready-made environment for developers interested in data science, machine learning and artificial intelligence, and to concentrate only on the tasks they are interested in.

Data Science Cloud interfaceThe interface of the Oracle Data Science Cloud service has the Jupyter Notebook interface that users are accustomed to and includes all the features in local installation.

Posted on

XGBoost: A Deep Dive Into Boosting

Every day we hear about the breakthroughs in artificial intelligence. However, have you wondered what challenges it faces?

Challenges occur in highly unstructured data like DNA sequencing, credit card transactions, and even in cybersecurity, which is the backbone of keeping our online presence safe from fraudsters. Does this thought make you yearn to know more about the science and reasoning behind these systems? Do not worry! We’ve got you covered. In the cyber era, machine learning (ML) has provided us with the solutions to these problems with the implementation of Gradient Boosting Machines (GBM). We have ample algorithms to choose from to do gradient boosting for our training data but still, we encounter different issues like poor accuracy, high loss, large variance in the result. 

Here, we are going to introduce you to a state of the art machine learning algorithm XGBoost built by Tianqi Chen, that will not only overcome the issues but also perform exceptionally well for regression and classification problems. This blog will help you discover the insights, techniques, and skills with XGBoost that you can then bring to your machine learning projects.

Posted on

Automatic Machine Learning (AutoML) Infrastructure — Oracle Data Science Cloud Service

In this article, I will talk about AutoML, one of the features that come with the Oracle Cloud Data Science Service, and I hope it will be a useful article in terms of awareness.

As it is known and mentioned in my previous articles, Oracle recently added a new service called Data Science to cloud services. This service has been offered to users as a platform where many libraries come pre-installed. This platform, which includes many features like prototype development, project development, model management, to the production of produced models, contains many new features. Undoubtedly, one of the most interesting and useful features is the AutoML feature.

Posted on

What’s Stopping the Democratization of AI?

With companies across industries waking up to the reality that adopting AI isn’t merely an option anymore, the question has shifted to how its adoption and implementation can be simplified. In other words, how does one break down the immensely tall barriers around the complicated world of AI and leverage the undeniable advantages it has to offer in terms of managing the scale and complexity of all the data that’s being gathered through the Internet of Things (IoT) already?

There’s no figment of doubt that it is indeed the need of the hour when every industry is fighting a losing battle with scale — the sheer magnitude of data streaming in from the millions (at times billions) of sensors, tools, and equipment.

Posted on

The ”O” Word: The Year of the Graph Newsletter: November 2019

How do you manage your enterprise data in order to keep track of it and be able to build and operate useful applications? This is a key question all data management systems are trying to address, and knowledge graphs, graph databases, and graph analytics are no different. What is different about knowledge graphs is that they may actually be the most elaborate and holistic way to manage your enterprise domain knowledge.

For people who have been into knowledge graphs or ontologies, as their original name was, this is old news. What is new is that more and more people today seem to be listening, rather than dismissing ontology as too complex, unrealistic, academic, etc. These last couple of months, we've seen a flurry of activity on all of these technologies. From organizational culture and adoption to events, research and tutorials, it's all here.

Posted on

Why Use SQL Over Excel

SQL is replacing Excel in many fields, and data analysis is certainly one of them. If you are still using Excel as a data analyst, you are missing something very valuable. SQL can make your life easier, as it’s more efficient and faster than Excel. How and from where can you learn SQL?

You may also like: The Unreasonable Effectiveness of SQL

Posted on

Two Rookie Mistakes to Avoid When Training a Predictive Model

Mistakes to avoid when training a predictive modelWhen creating predictive models, it's important to measure accuracy to be able to clearly articulate how good the model is. This article talks about two mistakes that are commonly made when measuring these accuracy values.

1. Measuring Accuracy on the Same Data Used for Training

One common mistake that gets made is measuring the accuracy of the same data that was trained. For example, say you have data from 2017 and 2018 for customer churn. Say you feed all that data to train the model and subsequently use the same data to predict and compare the predictions with the actual results. That is like you are given a question paper before the exam to study at home and the exact same question paper was given to you the next day in the exam. Obviously, that person is going to do great in the exam.