Xebia

APRIL 2, 2024

Selecting the right architectural serving pattern is paramount in creating the most business value from your model. In this blog we will discuss the most common serving architectures 1 ; batch predicting, on-demand synchronous serving and streaming serving. The process is triggered periodically. It doesn’t have to be.

Machine Learning 162

Machine Learning Artificial Inteligence Architecture Data 162

Xebia

OCTOBER 5, 2023

In the realm of DevOps, a smooth CI/CD pipeline is more than a convenience — it’s the backbone of efficient software delivery. But are you leveraging GitLab’s pipeline capabilities to the fullest? Whether you’re a newbie or a seasoned expert, this blog hopes to introduce some tips that can turbocharge your pipelines.

Testing 130

Testing Examples DevOps Continuous Delivery 130

Xebia

MAY 16, 2023

Dbt streamlines the analytics engineering workflow, covering all steps from development and testing to documentation and deployment, all in a non-intrusive way. In my previous blog post, I discussed how to manage multiple BigQuery projects with one dbt Cloud project , but left the setup of the deployment pipeline for a later moment.

Cloud 130

Cloud Azure DevOps Analytics 130

Xebia

JULY 6, 2023

Recently a customer reported several incidents with a data pipeline, causing the business to complain about the lack of visibility on the data transfer processes. To sort this, checks and balances are added to data pipeline steps to identify the bottleneck, and enable a single pane of glass. Check out this blog.

Google Cloud 130

Google Cloud Windows Cloud How To 130

Netflix Tech

NOVEMBER 14, 2023

Many metrics in Netflix’s financial reports are powered and reconciled with efforts from our team! In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Given our role on this critical path, accuracy is paramount. Let’s dive in!

Data Engineering 105

Data Engineering Engineering Data Systems Review 105

Xebia

JANUARY 30, 2023

Think of things like deployment pipelines for your applications. In this blog post I will guide you to how I do it for my pet projects. We only need 1 repository and a pipeline to deploy our infrastructure. But we also need a pipeline that deploys our feature branch. You can share these resources across your pipelines.

Applications 130

Applications AWS Resources Policies 130

Xebia

MARCH 19, 2023

In this blog I will show you how to create and deploy a Slackbot on Google Cloud Platform in less than 5 minutes using copier template. But that is the easy part: you also need to configure the application in Slack, deploy it so that is available 7×24, write unit tests and create CI/CD deployment pipelines and more.

Google Cloud 130

Google Cloud Cloud How To Applications 130

Xebia

AUGUST 23, 2023

Using AWS Glue in your data processing pipelines can be really powerful. An example of this is where we ran into some issues with a data processing pipeline that was unable to handle schema changes in the incoming data. The pipeline was working fine but then all of a sudden it stopped working. showString. in stage 1.0 (TID

AWS 130

AWS Software Review Systems Review Data 130

DevOps.com

JULY 27, 2022

CircleCI today announced it has integrated its continuous integration/continuous delivery (CI/CD) platform with a rival software-as-a-service (SaaS) offering from GitLab. DevOps teams can now, for example, trigger pipelines in CircleCI from GitLab forked merge requests in addition to merging requests and making commits for specific branches.

Continuous Delivery 128

Continuous Delivery Continuous Integration DevOps Testing 128

Flexagon

SEPTEMBER 22, 2022

We have recently published a number of blogs on these new features, but you may be wondering what else is new and coming with FlexDeploy 6.0. Define and store your tests along with your source code and utilize continuous testing in your release pipelines. Assess quality metrics and thresholds and trigger actions based on results.

UI/UX 78

UI/UX Software Review Metrics Testing 78

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. Until now, the setup of such pipelines still required knowledge of Airflow and the associated python configurations. Easing development friction.

Data Engineering 120

Data Engineering Engineering Data Virtualization 120

Xebia

AUGUST 15, 2023

Data contracts fundamentals Imagine a scenario where a data team develops a pipeline to extract, transform, and load customer data from various sources into a centralized data warehouse. From the release of v1.5 , and more recently v1.6 , dbt started to focus on multi-project features , as project sizes and complexity are growing fast.

Data 130

Data Software Review Testing Cloud 130

Xebia

APRIL 18, 2023

Did you ever ran into the scenario where you wanted to update files in a repository from your pipeline? Updating documentation In one of my previous blogs I wrote on how you can manage diagrams as code. The approaches that I often is that you tag the container as latest and trigger a redeployment. ssh mkdir ~/.ssh

How To 130

How To Examples Tools AWS 130

CircleCI

JANUARY 24, 2022

I will show you how to use scheduled pipelines to run the security tests on a regular schedule. You can clone the code for the form from this repository by running this command from anywhere on your system: git clone -b base-project --single-branch [link]. Adding the pipeline configuration script. Prerequisites.

Software Review 97

Software Review Technical Review Testing Systems Review 97

Netflix Tech

NOVEMBER 20, 2023

It also improves the engineering productivity by simplifying the existing pipelines and unlocking the new patterns. In this blog post, we talk about the landscape and the challenges in workflows at Netflix. It must be aware of the change and can capture the changes from the source table(s) and then keep tracking those changes.

Windows 84

Windows Software Review Data Engineering 84

Prisma Clud

SEPTEMBER 14, 2023

Get an in-depth look at the attack vectors, technical details and a real-world demo in this blog post highlighting our latest research. The payload tries to steal secrets or create a reverse shell, whether running in pipelines or production environments. But how can the attackers extend their reach and infect more repositories?

Malware 144

Malware Open Source Research Software 144

Xebia

JULY 13, 2023

You can invalidate the cache but objects like photos and videos benefit from this cache. So for this blog post that would result to: $ git diff-tree --no-commit-id --name-only -r HEAD content/blog/2023/07/invalidate-cloudfront-cache-for-hugo-websites/index.md This is because you will most likely end up on the same edge server.

Lambda 130

Lambda AWS Video Tools 130

Dzone - DevOps

FEBRUARY 16, 2021

In our previous article ( [link] ), we’ve seen the basics and how to create a pipeline. In this particular blog, we will see how to extend the basic pipeline which was created in the previous blog by creating a release pipeline that will help us not only automate a deployment but complete the release.

Azure 81

Azure DevOps Software How To 81

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. Data pipelines are composed of multiple steps with dependencies and triggers. Modernizing pipelines.

Data Engineering 118

Data Engineering Technical Review Software Review Engineering 118

Xebia

DECEMBER 9, 2023

Click to read the PDF version GitHub Actions [1] are a powerful way of creating a pipeline to act on events in GitHub. By creating a workflow file you run actions on code updates to build your application, automate triaging tasks from issues, and loads of other helpful uses. Let’s start with an introduction to GitHub Actions first.

Software Review 130

Software Review DevOps Examples Engineering 130

Flexagon

MARCH 21, 2023

In this blog, we’ll cover how to integrate PMD for Java, HTML, and PLSQL into your release pipelines in FlexDeploy. After a quick configuration, you’ll be ready to execute PMD scans as part of your pipeline in FlexDeploy. You can insert the scan before a build if you are sourcing files from SCM.

Software Review 78

Software Review Report Analysis Programming 78

Flexagon

JULY 15, 2022

If you are looking for an introduction to FlexDeploy’s support for Salesforce, start with this blog series. In this blog, we’ll cover how to integrate PMD source code analyzer for apex classes, visual force pages, and JavaScript into your release pipelines. You can also download the report from this tab.

Software Review 78

Software Review Report DevOps Analysis 78

Perficient

OCTOBER 31, 2023

We will go over the advantages of the Azure DevOps connection with Katalon in this blog post and walk you through the setup process. CI/CD Integration Autonomous testing interfaces with CI/CD methods to effortlessly fit into contemporary software development pipelines. Create a new Pipeline. Choose Starter pipeline.

Azure 52

Azure DevOps Testing Software Development 52

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. Data Reprocessing Pipeline Flow Figure 1.

Data 94

Data Metrics Machine Learning Artificial Inteligence 94

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. Impala vs Spark Use Impala primarily for analytical workloads triggered by end users. So which open source pipeline tool is better, NiFi or Airflow?

Tools 76

Tools Data Engineering Analytics Testing 76

Perficient

JUNE 12, 2024

What is CI/CD? A CI/CD (Continuous Integration/Continuous Deployment) pipeline is an automated workflow or series of steps that developers use to build, test, and deploy their code changes. This blog goes into more detail about the reference implementation and provides example code to assist with it. Why Do We Need CI/CD?

Continuous Integration 52

Continuous Integration Test-Driven Development Testing Software Development 52

Perficient

FEBRUARY 14, 2024

AEM as a Cloud Service provides the means to isolate UI design along with deployment pipelines dedicated to them. I encourage you to walk through this to set up your own front-end design project and deployment pipeline. In our dev author, I also created a test app using the site from template option in AEM Sites.

Testing 52

Testing Cloud Games Development 52

CircleCI

DECEMBER 16, 2022

Fortunately, CircleCI provides several tools you can use to streamline the onboarding process and get up and running with a fast and secure pipeline in a matter of minutes, regardless of experience level. You can select which projects, branches, and statuses to follow from the settings pane. Config SDK. Config policy management.

Tools 59

Tools Software Review Technical Review Policies 59

Cloudera

JULY 13, 2021

CDP Data Engineering offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual profiling, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. . A key aspect of ETL or ELT pipelines is automation.

Data Engineering 127

Data Engineering Azure Engineering Enterprise 127

CircleCI

MARCH 22, 2022

Add scheduled pipelines to your config. Define scheduled triggers using project settings. In this tutorial, I will show you how you can effectively generate and send an invoice to your client using the Invoice generator API and scheduled pipelines from CircleCI. Prerequisites. For this tutorial, you will need: Node.js

Applications 52

Applications Technical Review Software Review Systems Review 52

Cloudera

SEPTEMBER 21, 2021

Many customers looking at modernizing their pipeline orchestration have turned to Apache Airflow, a flexible and scalable workflow manager for data engineers. Set up a connection to a CDE Jobs API in your Airflow as follows: # Create connection from the CLI (can also be done from the UI): #Airflow 2.x: conn-type 'hive'.

Off-The-Shelf 99

Off-The-Shelf Data Engineering Virtualization Cloud 99

CircleCI

JANUARY 13, 2022

This new access opens up a number of new possibilities for your pipelines. We will be using the sample orb I built, which contains a command and a job that lets you trigger CircleCI pipelines in other projects using cURL. Understanding of how pipelines work. We created 2 scripts: A commands/trigger-pipeline.yml command.

Organization 52

Organization Software Review Testing Hardware 52

Flexagon

APRIL 18, 2022

Here is what a package would look like: Integrations can be discovered from OIC itself or from a source control management system like Git. Here is what the files screen looks like after discovering from SCM: Why is this such a big deal? For more info on partial deployments check out this blog. Populate Trigger.

Cloud 78

Cloud DevOps Development Examples 78

Cloudera

JUNE 16, 2022

Organizations are increasingly building low-latency, data-driven applications, automations, and intelligence from real-time data streams. A number of large financial services companies use CSP to power their global fraud processing pipelines and prevent users from exploiting race conditions in the loan approval process. .

Data 88

Data Disaster Recovery Analytics Microservices 88

Prisma Clud

JULY 13, 2023

Many organizations using this architecture allow their CI/CD environment to receive webhook events from the SaaS source control vendors to trigger pipeline jobs. Let’s walk through how we abused repository webhooks and then discuss practical ways you can protect yourself from this CI/CD security risk. Let’s discover how.

System 52

System Firewall Authentication Internet 52

Xebia

JANUARY 27, 2023

From my experience I can say that while Cypress has a speedy testrunner, there can still be reasons why your tests run slow. Renovate or Dependabot , which takes care of updating your Cypress dependencies within your package.json as well as the Cypress Docker image you should be using in your pipeline definition file. What do I see ?

Testing 130

Testing Cloud Metrics Examples 130

MagmaLabs

JUNE 3, 2021

Reading Time: 3 minutes A Pipeline is a collection of apps relating to a single codebase. This is a blog post on how to get the most out of your available tools and ensuring your review apps will remain in Heroku’s free tier. Promoting a slug does not trigger a new build, so choose carefully. Creating a Pipeline.

Technical Review 52

Technical Review How To Testing Agile 52