Expert insights. Personalized for you.
The O’Reilly Data Show Podcast: A special episode to mark the 100th episode. This episode of the Data Show marks our 100th episode. We had a collection of friends who were key members of the data science and big data communities on hand and we decided to record short conversations with them. The logistics of studio interviews proved too complicated, but those Foo Camp conversations got us thinking about starting a podcast, and the Data Show was born. MORE
Big data can be quite a confusing concept to grasp. What to consider big data and what is not so big data? Big data is still data, of course. But it requires a different engineering approach and not just because of its amount. Regular data processing. MORE
To mix the power of the data and the importance of people to offer business intelligence is a key point nowadays. Machine Learning Data Analytics Agile Culture Startup Innovation Culture Culture & Methods AI, ML & Data Engineering article MORE
Being at the top of data science capabilities, machine learning and artificial intelligence are buzzing technologies many organizations are eager to adopt. However, they often forget about the fundamental work – data literacy, collection, and infrastructure – that must be done prior to building intelligent data products. Data science layers towards AI, Source: Monica Rogati. Explaining Data Engineering and Data Warehouse. MORE
It’s important to understand the differences between a data engineer and a data scientist. Misunderstanding or not knowing these differences are making teams fail or underperform with big data. I think some of these misconceptions come from the diagrams that are used to describe data scientists and data engineers. Overly simplistic venn diagram with data scientists and data engineers. Yes, both positions work on big data. MORE
The O’Reilly Data Show Podcast: Avner Braverman on what’s missing from serverless today and what users should expect in the near future. In this episode of the Data Show , I spoke with Avner Braverman , co-founder and CEO of Binaris , a startup that aims to bring serverless to web-scale and enterprise applications. Continue reading What data scientists and data engineers can do with current generation serverless technologies MORE
Senior Software Engineer – Big Data. IO is the global leader in software-defined data centers. IO has pioneered the next-generation of data center infrastructure technology and Intelligent Control, which lowers the total cost of data center ownership for enterprises, governments, and service providers. We are looking for a talented Big Data Software Engineer to join the Applied Intelligence group in San Francisco. By Bob Gourley. MORE
The former sees growing investment in data analytics to become data-driven (45% of organizations expect to increase their spending in this area) while the latter is fueled by disruptive technology and the adoption of AI (41% of organizations name it as their game changer). Both rely virtually entirely on the enterprise leveraging of data. They need a range of analytical capabilities from data engineering to data warehousing to operational databases and data science. MORE
I’m interrupting the regular programming for a quick announcement: we’re looking for data engineers at Better. Migrate our data warehouse to Redshift. Write and productionize a web scraper to ingest a bunch of financial third party data. Fit Gamma distributions to conversion data to understand the time lag and conversion rates. This position is very engineering-heavy at its core, and the main qualification is solid programming skills. MORE
Is it all about customer-centric business, or is there any data left? Can we integrate data analytics and customer empathy? Customers & Requirements Big Data Data Analytics Project Management Database Data Analysis Infrastructure Surveys Agile Culture & Methods AI, ML & Data Engineering article MORE
The business value of applying data science in organizations is incontestable. Data science work can be divided into analytical and data preparation work. Examples of data preparation activities. MORE
With the rise of big data and data science, storage and retrieval have become a critical pipeline component for data use and analysis. Recently, new data storage technologies have emerged. Which one is best suited for data engineering? In this article, Toptal Data Scientist Ken Hu compares three prominent storage technologies within the context of data engineering MORE
Or, why science and engineering are still different disciplines. "A He would have to ask an engineer to do it for him.". A few months ago, I wrote about the differences between data engineers and data scientists. An interesting thing happened: the data scientists started pushing back, arguing that they are, in fact, as skilled as data engineers at data engineering. Otherwise, this leads to failure with big data projects. MORE
Automated testing Artificial Intelligence Programming Automation Development Culture & Methods AI, ML & Data Engineering articleThese days AI is a big buzzword. While it rises in popularity, the controversy surrounding it flourishes as well. MORE
For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance. MORE
In particular, there has been a significant increase in demand for data scientists. Companies are searching and competing for increasingly scarce data scientists as the […]. Artificial Intelligence Big Data and Analytics Cloud Computing CTO artificial intelligence big data data data engineer data scientist Enterprise MORE
Developing ML with agile has a few challenges that new teams coming up in the space need to be prepared for - from new roles like Data Scientists to concerns in reproducibility and dependency management. Machine learning is a powerful new tool, but how does it fit in your agile development? MORE
Between growing data and analytics teams, more collaborative projects, and GDPR , security and governance are becoming bigger issues. As organizations' analytical maturity improves, it's increasingly critical that business and technical users alike understand data best practices. Hadoop Technology Data Engineering data architecture MORE
As AI increasingly gains popularity among enterprises, companies are actively seeking data scientists who possess data science skills. Many enterprises confuse the roles of data scientists and data engineers. Artificial Intelligence Big Data and Analytics CTOEven though some traits, skills, programming languages and tools are shared by both roles, the overall roles and core skill sets are different and are not [.]. MORE
A traditional BI and analytics organization consists of three main groups: Analysts that develop reports often using sample data. The data management team – modelers that take requests, find data, and develop models to answer the questions. In a big data world, we often see three new roles emerge and work more closely together: data engineers, data scientists and architects. You can think of them as the data workhorse. MORE
In part 1 of this series we introduced Kentik Data Engine™, the backend to Kentik Detect™, which is a large-scale distributed datastore that is optimized for querying IP flow records (NetFlow v5/9, sFlow, IPFIX) and related network data (GeoIP, BGP, SNMP). In this query we’ve grouped the data by source ? Time-series data Summary tables are great, but often we want time-series data to build visualizations. Want to try KDE with your own network data? MORE
I’m interrupting the regular programming for a quick announcement: we’re looking for data engineers at Better. Migrate our data warehouse to Redshift. Write and productionize a web scraper to ingest a bunch of financial third party data. Fit Gamma distributions to conversion data to understand the time lag and conversion rates. This position is very engineering-heavy at its core, and the main qualification is solid programming skills. MORE
Here at Kentik, we’ve applied many of the same concepts to Kentik Data Engine™ (KDE), a datastore optimized for querying IP flow records (NetFlow v5/9, sFlow, IPFIX) and related network data (GeoIP, BGP, SNMP). KDE is the backend of Kentik Detect™ and as such enables users to query network data and view visualizations via the Kentik portal, a fast, intuitive UI. Next, let’s look at capacity: how big is our “big data”? MORE
I mentioned in an earlier blog titled, “Staffing your big data team, ” that data engineers are critical to a successful data journey. That said, most companies that are early in their journey lack a dedicated engineering group. And the longer it takes to put a team in place, the likelier it is that your big data project will stall. However, it’s imperative to find people who have an intense interest in the data that they are working with. MORE
However, data engineering can become a major constraint within that process. data science continuous delivery data analytics data analysis data engineering data engineer continuous engineering continuous delivery foundationModern DevOps practices of continuous testing, integration, deployment/delivery, and monitoring form the backbone of a smooth deployment pipeline that continuously feeds back into itself for improvement. MORE
David Schaaf explains how data science and data engineering can work together to deliver results to decision makers. Continue reading Jupyter notebooks and the intersection of data science and data engineering MORE
The post Data Engineering: The Heavy Lifting Behind IoT appeared first on QBurst - Blog. This post is part of our continuing blog series on the Internet of Things. In our previous posts, we discussed sensors, wireless technologies in IoT, and Connected Operations: 3 IoT Scenarios. Smart cities, self-driving cars, intelligent machines—the IoT market is exploding with “Things.” The ease with which they cross over from sci-fi to real life […]. MORE
What is Cloudera Data Engineering (CDE) ? Cloudera Data Engineering is a serverless service for Cloudera Data Platform (CDP) that allows you to submit jobs to auto-scaling virtual clusters. The Cloudera Data Engineering service API is documented in Swagger. MORE
quintillion bytes of data generated daily, data scientists get busier than ever. And data science provides us with methods to make use of this data. So, along with data scientists who create algorithms, there are data engineers, the architects of data platforms. MORE
and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley. MORE
But there is far less agreement on what that term “data analytics” actually means?—?or Even within Netflix, we have many groups that do some form of data analysis, including business strategy and consumer insights. When you think about data at Netflix, what comes to mind? MORE
While we know that data engineers are very different than data architects — as the latter conceptualize data frameworks and the former build and maintain them — the data engineer function has evolved quite a bit in recent years. Data Basics Featured MORE
Get a basic overview of data engineering and then go deeper with recommended resources. As the the data space has matured, data engineering has emerged as a separate and related role that works in concert with data scientists. Continue reading Data engineering: A quick and simple definition MORE
Use Cases Project Management Book Review Artificial Intelligence Machine Learning Programming InfoQ Training / Certification Agile Development Culture & Methods AI, ML & Data Engineering article MORE
Read Andrew Wooler explain how Kinetica can provide a cost-effective streaming data warehouse on Forbes : I recently watched the movie Ford v Ferrari, based on the true story of […]. MORE
To become a machine learning engineer, you have to interview. interviewing An introduction to Machine Learning Machine Learning Culture & Methods AI, ML & Data Engineering articleYou have to gain relevant skills from books, courses, conferences, and projects. MORE
Data from the US Treasury website show which companies received PPP loans and how many jobs were retained. Analysis of this data presents three challenges. First, the size of the data is significant. Cloudera Data Engineering (CDE). MORE
QCon returned to London this past March for its fourteenth year in the city, attracting over 1,600 senior developers, architects, data engineers, team leads, and CTOs. QCon London 2020 DevOps Development Architecture & Design Culture & Methods AI, ML & Data Engineering article MORE
Microservices Chaos Engineering Edge Computing Distributed Systems Azure Continuous Delivery AWS Google Cloud Data Mesh Kubernetes Modular Monolith Jakarta EE Culture & Methods Development Architecture & Design AI, ML & Data Engineering DevOps article MORE
Programming Languages System Programming Programming DevOps Culture & Methods Development Architecture & Design AI, ML & Data Engineering articleAt the end of 2019, InfoQ ran a survey of our readers to find out what tools, techniques, and languages they were using. MORE
Data Engineering Related Topics O'Reilly Media - Data
APRIL 11, 2018
It’s important to understand the differences between a data engineer and a data scientist. Misunderstanding or not knowing these differences are making teams fail or underperform with big data. I think some of these misconceptions come from the diagrams that are used to describe data scientists and data engineers. Overly simplistic venn diagram with data scientists and data engineers. Yes, both positions work on big data.
O'Reilly Media - Data
JULY 16, 2018
Get a basic overview of data engineering and then go deeper with recommended resources. As the the data space has matured, data engineering has emerged as a separate and related role that works in concert with data scientists. Continue reading Data engineering: A quick and simple definition
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Cloudera
APRIL 14, 2021
What is Cloudera Data Engineering (CDE) ? Cloudera Data Engineering is a serverless service for Cloudera Data Platform (CDP) that allows you to submit jobs to auto-scaling virtual clusters. The Cloudera Data Engineering service API is documented in Swagger.
Dataiku
FEBRUARY 9, 2021
While we know that data engineers are very different than data architects — as the latter conceptualize data frameworks and the former build and maintain them — the data engineer function has evolved quite a bit in recent years. Data Basics Featured
Speaker: Mindy Chen, Director of Decision Science, Hudl
In this webinar, we will unpack how data team structures have evolved by drawing on examples from our customers at Snowplow and discussing the pros and cons of the different structures that we have seen. We will be joined by Mindy Chen, Director of Decision Science at Hudl, who will take us on a journey through the challenges and opportunities during her experience of growing her data team from 3 to 20.
CTOvision
DECEMBER 15, 2020
Read Andrew Wooler explain how Kinetica can provide a cost-effective streaming data warehouse on Forbes : I recently watched the movie Ford v Ferrari, based on the true story of […].
Erik Bernhardsson
JANUARY 27, 2018
I’m interrupting the regular programming for a quick announcement: we’re looking for data engineers at Better. Migrate our data warehouse to Redshift. Write and productionize a web scraper to ingest a bunch of financial third party data. Fit Gamma distributions to conversion data to understand the time lag and conversion rates. This position is very engineering-heavy at its core, and the main qualification is solid programming skills.
O'Reilly Media - Data
MAY 24, 2018
The O’Reilly Data Show Podcast: A special episode to mark the 100th episode. This episode of the Data Show marks our 100th episode. We had a collection of friends who were key members of the data science and big data communities on hand and we decided to record short conversations with them. The logistics of studio interviews proved too complicated, but those Foo Camp conversations got us thinking about starting a podcast, and the Data Show was born.
Altexsoft
JUNE 25, 2019
Being at the top of data science capabilities, machine learning and artificial intelligence are buzzing technologies many organizations are eager to adopt. However, they often forget about the fundamental work – data literacy, collection, and infrastructure – that must be done prior to building intelligent data products. Data science layers towards AI, Source: Monica Rogati. Explaining Data Engineering and Data Warehouse.
Altexsoft
AUGUST 25, 2020
Big data can be quite a confusing concept to grasp. What to consider big data and what is not so big data? Big data is still data, of course. But it requires a different engineering approach and not just because of its amount. Regular data processing.
O'Reilly on Data
APRIL 9, 2019
Or, why science and engineering are still different disciplines. "A He would have to ask an engineer to do it for him.". A few months ago, I wrote about the differences between data engineers and data scientists. An interesting thing happened: the data scientists started pushing back, arguing that they are, in fact, as skilled as data engineers at data engineering. Otherwise, this leads to failure with big data projects.
|
Cloudera
JANUARY 12, 2018
I mentioned in an earlier blog titled, “Staffing your big data team, ” that data engineers are critical to a successful data journey. That said, most companies that are early in their journey lack a dedicated engineering group. And the longer it takes to put a team in place, the likelier it is that your big data project will stall. However, it’s imperative to find people who have an intense interest in the data that they are working with.
Cloudera
SEPTEMBER 17, 2020
For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance.
Altexsoft
APRIL 22, 2020
quintillion bytes of data generated daily, data scientists get busier than ever. And data science provides us with methods to make use of this data. So, along with data scientists who create algorithms, there are data engineers, the architects of data platforms.
Cloudera
AUGUST 20, 2020
Data from the US Treasury website show which companies received PPP loans and how many jobs were retained. Analysis of this data presents three challenges. First, the size of the data is significant. Cloudera Data Engineering (CDE).
CTOvision
APRIL 27, 2018
In particular, there has been a significant increase in demand for data scientists. Companies are searching and competing for increasingly scarce data scientists as the […]. Artificial Intelligence Big Data and Analytics Cloud Computing CTO artificial intelligence big data data data engineer data scientist Enterprise
QBurst
FEBRUARY 12, 2019
The post Data Engineering: The Heavy Lifting Behind IoT appeared first on QBurst - Blog. This post is part of our continuing blog series on the Internet of Things. In our previous posts, we discussed sensors, wireless technologies in IoT, and Connected Operations: 3 IoT Scenarios. Smart cities, self-driving cars, intelligent machines—the IoT market is exploding with “Things.” The ease with which they cross over from sci-fi to real life […].
O'Reilly on Data
APRIL 11, 2019
The O’Reilly Data Show Podcast: Avner Braverman on what’s missing from serverless today and what users should expect in the near future. In this episode of the Data Show , I spoke with Avner Braverman , co-founder and CEO of Binaris , a startup that aims to bring serverless to web-scale and enterprise applications. Continue reading What data scientists and data engineers can do with current generation serverless technologies
CTOvision
MAY 1, 2014
Senior Software Engineer – Big Data. IO is the global leader in software-defined data centers. IO has pioneered the next-generation of data center infrastructure technology and Intelligent Control, which lowers the total cost of data center ownership for enterprises, governments, and service providers. We are looking for a talented Big Data Software Engineer to join the Applied Intelligence group in San Francisco. By Bob Gourley.
Toptal
DECEMBER 6, 2016
With the rise of big data and data science, storage and retrieval have become a critical pipeline component for data use and analysis. Recently, new data storage technologies have emerged. Which one is best suited for data engineering? In this article, Toptal Data Scientist Ken Hu compares three prominent storage technologies within the context of data engineering
O'Reilly on Data
AUGUST 27, 2018
David Schaaf explains how data science and data engineering can work together to deliver results to decision makers. Continue reading Jupyter notebooks and the intersection of data science and data engineering
Kentik
MAY 2, 2016
In part 1 of this series we introduced Kentik Data Engine™, the backend to Kentik Detect™, which is a large-scale distributed datastore that is optimized for querying IP flow records (NetFlow v5/9, sFlow, IPFIX) and related network data (GeoIP, BGP, SNMP). In this query we’ve grouped the data by source ? Time-series data Summary tables are great, but often we want time-series data to build visualizations. Want to try KDE with your own network data?
Kentik
APRIL 25, 2016
Here at Kentik, we’ve applied many of the same concepts to Kentik Data Engine™ (KDE), a datastore optimized for querying IP flow records (NetFlow v5/9, sFlow, IPFIX) and related network data (GeoIP, BGP, SNMP). KDE is the backend of Kentik Detect™ and as such enables users to query network data and view visualizations via the Kentik portal, a fast, intuitive UI. Next, let’s look at capacity: how big is our “big data”?
InfoQ Articles
FEBRUARY 25, 2021
Developing ML with agile has a few challenges that new teams coming up in the space need to be prepared for - from new roles like Data Scientists to concerns in reproducibility and dependency management. Machine learning is a powerful new tool, but how does it fit in your agile development?
Dzone - DevOps
OCTOBER 20, 2020
However, data engineering can become a major constraint within that process. data science continuous delivery data analytics data analysis data engineering data engineer continuous engineering continuous delivery foundationModern DevOps practices of continuous testing, integration, deployment/delivery, and monitoring form the backbone of a smooth deployment pipeline that continuously feeds back into itself for improvement.
InfoQ Culture Methods
JULY 28, 2020
To become a machine learning engineer, you have to interview. interviewing An introduction to Machine Learning Machine Learning Culture & Methods AI, ML & Data Engineering articleYou have to gain relevant skills from books, courses, conferences, and projects.
CTOvision
JUNE 7, 2018
As AI increasingly gains popularity among enterprises, companies are actively seeking data scientists who possess data science skills. Many enterprises confuse the roles of data scientists and data engineers. Artificial Intelligence Big Data and Analytics CTOEven though some traits, skills, programming languages and tools are shared by both roles, the overall roles and core skill sets are different and are not [.].
InfoQ Culture Methods
JULY 17, 2020
To mix the power of the data and the importance of people to offer business intelligence is a key point nowadays. Machine Learning Data Analytics Agile Culture Startup Innovation Culture Culture & Methods AI, ML & Data Engineering article
InfoQ Culture Methods
MAY 4, 2020
Automated testing Artificial Intelligence Programming Automation Development Culture & Methods AI, ML & Data Engineering articleThese days AI is a big buzzword. While it rises in popularity, the controversy surrounding it flourishes as well.
InfoQ Culture Methods
APRIL 7, 2020
QCon returned to London this past March for its fourteenth year in the city, attracting over 1,600 senior developers, architects, data engineers, team leads, and CTOs. QCon London 2020 DevOps Development Architecture & Design Culture & Methods AI, ML & Data Engineering article
Data Virtualization
OCTOBER 14, 2020
The business value of applying data science in organizations is incontestable. Data science work can be divided into analytical and data preparation work. Examples of data preparation activities.
InfoQ Culture Methods
MARCH 19, 2020
Use Cases Project Management Book Review Artificial Intelligence Machine Learning Programming InfoQ Training / Certification Agile Development Culture & Methods AI, ML & Data Engineering article
InfoQ Articles
DECEMBER 28, 2020
Microservices Chaos Engineering Edge Computing Distributed Systems Azure Continuous Delivery AWS Google Cloud Data Mesh Kubernetes Modular Monolith Jakarta EE Culture & Methods Development Architecture & Design AI, ML & Data Engineering DevOps article
The Netflix TechBlog
SEPTEMBER 18, 2020
But there is far less agreement on what that term “data analytics” actually means?—?or Even within Netflix, we have many groups that do some form of data analysis, including business strategy and consumer insights. When you think about data at Netflix, what comes to mind?
InfoQ Culture Methods
APRIL 18, 2020
Programming Languages System Programming Programming DevOps Culture & Methods Development Architecture & Design AI, ML & Data Engineering articleAt the end of 2019, InfoQ ran a survey of our readers to find out what tools, techniques, and languages they were using.
Cloudera
JANUARY 3, 2018
A traditional BI and analytics organization consists of three main groups: Analysts that develop reports often using sample data. The data management team – modelers that take requests, find data, and develop models to answer the questions. In a big data world, we often see three new roles emerge and work more closely together: data engineers, data scientists and architects. You can think of them as the data workhorse.
The Netflix TechBlog
SEPTEMBER 18, 2020
and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley.
InfoQ Culture Methods
SEPTEMBER 6, 2019
Is it all about customer-centric business, or is there any data left? Can we integrate data analytics and customer empathy? Customers & Requirements Big Data Data Analytics Project Management Database Data Analysis Infrastructure Surveys Agile Culture & Methods AI, ML & Data Engineering article
Dataiku
AUGUST 9, 2019
Between growing data and analytics teams, more collaborative projects, and GDPR , security and governance are becoming bigger issues. As organizations' analytical maturity improves, it's increasingly critical that business and technical users alike understand data best practices. Hadoop Technology Data Engineering data architecture
Cloudera
FEBRUARY 11, 2019
The former sees growing investment in data analytics to become data-driven (45% of organizations expect to increase their spending in this area) while the latter is fueled by disruptive technology and the adoption of AI (41% of organizations name it as their game changer). Both rely virtually entirely on the enterprise leveraging of data. They need a range of analytical capabilities from data engineering to data warehousing to operational databases and data science.
128 
Let's personalize your content