Insights: apache/airflow

69 Pull requests merged by 31 people

• `AirbyteHook` add cancel job option

  #24593 merged Jun 29, 2022

• Use our yaml util in all providers

  #24720 merged Jun 29, 2022

• Address all yarn test warnings

  #24722 merged Jun 29, 2022

• Fix migration 0080_2_0_2 - Replace null values before setting column not null

  #24585 merged Jun 29, 2022

• Grid task log, multi select and ts files migration.

  #24623 merged Jun 28, 2022

• Do not fail requeued TIs

  #23846 merged Jun 28, 2022

• Merge-friendly output of command hashes for breeze

  #24711 merged Jun 28, 2022

• Script to filter candidates for PR of the month based on heuristics

  #24654 merged Jun 28, 2022

• Fix unnecessary check for ARM images

  #24718 merged Jun 28, 2022

• Add AWS operators to create and delete RDS Database

  #24099 merged Jun 28, 2022

• Debounce status highlighting in Grid view

  #24710 merged Jun 28, 2022

• Add ARM image building for regular PRs

  #24664 merged Jun 28, 2022

• Protect against using "hook_params" in "get_hook" until we move to 2.3+

  #24706 merged Jun 28, 2022

• fix document about response_check in HttpSensor

  #24708 merged Jun 28, 2022

• Remove `hook-class-names` from provider.yaml

  #24702 merged Jun 28, 2022

• Fix Grid vertical scrolling

  #24684 merged Jun 28, 2022

• Add `airflow_kpo_in_cluster` label to KPO pods

  #24658 merged Jun 28, 2022

• Don't render child rows for closed groups

  #24637 merged Jun 28, 2022

• Remove "Label when approved" workflow

  #24704 merged Jun 28, 2022

• Migrate Google example DAG s3_to_gcs to new design AIP-47

  #24641 merged Jun 28, 2022

• Add batch option to `SqsSensor`

  #24554 merged Jun 28, 2022

• `WebHDFSHook` Bugfix/optional port

  #24550 merged Jun 28, 2022

• Add test_connection method to Databricks hook

  #24617 merged Jun 28, 2022

• Clear next method when clearing TIs

  #23929 merged Jun 28, 2022

• Rename 'resources' arg in Kub op to k8s_resources

  #24673 merged Jun 28, 2022

• Clean up task decorator type hints and docstrings

  #24667 merged Jun 28, 2022

• Remove commented out code

  #24685 merged Jun 28, 2022

• TriggerDagRunOperator.operator_extra_links is attr

  #24676 merged Jun 28, 2022

• Keep grid log scrolling limited to its wrapper div

  #24689 merged Jun 27, 2022

• Updated Snowflake provider connection documentation

  #24573 merged Jun 27, 2022

• Pass worker annotations to generated pod_template_file

  #24647 merged Jun 27, 2022

• Add dataset model

  #24613 merged Jun 27, 2022

• Add Core SQL Provider

  #24476 merged Jun 27, 2022

• Turn inability of creating mypy-cache into warning

  #24677 merged Jun 27, 2022

• Fix behaviour of build/pull after recent Breeze changes

  #24657 merged Jun 27, 2022

• S3ToSnowflakeOperator: escape single quote in s3_keys

  #24607 merged Jun 26, 2022

• Fix handling of GitHub Event types for new selective checks

  #24665 merged Jun 26, 2022

• Handle "workflow_run" event properly in selective-check

  #24656 merged Jun 26, 2022

• Use target commit SHA for build image workflow

  #24659 merged Jun 25, 2022

• Change Postgres auth method in docker-compose

  #24640 merged Jun 25, 2022

• Migrate Google example DAG bigquery_transfer to new design AIP-47

  #24543 merged Jun 25, 2022

• Remove selective checks from the "release workflow"

  #24655 merged Jun 25, 2022

• Remove misleading message from CI

  #24650 merged Jun 25, 2022

• Cleanup references to selective checks

  #24649 merged Jun 25, 2022

• Switch to new selective-checks in label-when-reviewed workflow

  #24651 merged Jun 25, 2022

• Convert selective checks to Breeze Python

  #24610 merged Jun 25, 2022

• Update short flags in breeze config command

  #24639 merged Jun 25, 2022

• Allows to specify different Python base image when building images

  #24634 merged Jun 24, 2022

• Fix timestamp defaults for sensorinstance

  #24638 merged Jun 24, 2022

• Have consistent types between the ORM and the migration files

  #24044 merged Jun 24, 2022

• ExternalTaskSensor respects soft_fail if the external task enters a failed_state

  #23647 merged Jun 24, 2022

• Add %z for %(asctime)s to fix timezone for logs on UI

  #24373 merged Jun 24, 2022

• Add --platform to breeze shell and start-airflow

  #24626 merged Jun 24, 2022

• Add gcp_conn_id argument to GoogleDriveToLocalOperator

  #24622 merged Jun 24, 2022

• Limit azure-servicebus to not be used on ARM

  #24635 merged Jun 24, 2022

• docs: sqlalchemy link update

  #24627 merged Jun 24, 2022

• update `task-generated mapping` example

  #23424 merged Jun 24, 2022

• We now need at least Flask-WTF 0.15

  #24621 merged Jun 23, 2022

• Upgrade FAB to 4.1.2

  #24619 merged Jun 23, 2022

• Lay the groundwork for migrating Airflow CLI to Rich+Click

  #24590 merged Jun 23, 2022

• Implement Azure Service Bus Queue Operators

  #24038 merged Jun 23, 2022

• Use sql_alchemy_conn for celery result backend when result_backend is not set

  #24496 merged Jun 23, 2022

• Fix usage of `click.get_terminal_size()`

  #24616 merged Jun 23, 2022

• Fix errors revealed on autoupgrade of breeze

  #24612 merged Jun 23, 2022

• Add note about image regeneration in June 2022

  #24524 merged Jun 23, 2022

• Fix recording console for new rich-click 1.5

  #24611 merged Jun 22, 2022

• Upgrade FAB to 4.1.1

  #24399 merged Jun 22, 2022

• Migrate jsx files that affect run/task selection to tsx

  #24509 merged Jun 22, 2022

• Update breeze mypy check documentation.

  #24609 merged Jun 22, 2022

25 Pull requests opened by 21 people

• doc: update concepts overview diagram

  #24601 opened Jun 22, 2022

• Implement Azure service bus subscription (create, delete)Operators

  #24625 opened Jun 24, 2022

• Rewrite the 'celery' command to use Click

  #24628 opened Jun 24, 2022

• Fix pid check

  #24636 opened Jun 24, 2022

• Convert Step Functions Example DAG to System Test (AIP-47)

  #24643 opened Jun 24, 2022

• Add @task.snowpark decorator

  #24652 opened Jun 25, 2022

• Add test_connection method to AWS hook

  #24662 opened Jun 25, 2022

• Adding generic SqlToSlackOperator

  #24663 opened Jun 26, 2022

• Add more selective provider tests

  #24666 opened Jun 26, 2022

• added component specific labels

  #24668 opened Jun 26, 2022

• Update AWS Connection docs and deprecate some extras

  #24670 opened Jun 26, 2022

• Move provider dependencies to inside provider folders

  #24672 opened Jun 27, 2022

• Warning for using listing conns & vars with secrets backend

  #24679 opened Jun 27, 2022

• Update release process for Providers

  #24680 opened Jun 27, 2022

• Add test_connection method to GoogleBaseHook

  #24682 opened Jun 27, 2022

• Use older version of node for pre-commit checks

  #24688 opened Jun 27, 2022

• WIP Task-level dataset deps

  #24691 opened Jun 28, 2022

• Add read-only REST API endpoint for Datasets

  #24696 opened Jun 28, 2022

• (GitHub CI) update most CI workflows (stale, build-images, etc)

  #24705 opened Jun 28, 2022

• fix connection extra parameter `auth_mechanism` in `HiveMetastoreHook` and `HiveServer2Hook`

  #24713 opened Jun 28, 2022

• Upgrade more javascript files to typescript

  #24715 opened Jun 28, 2022

• KubernetesPodOperator can get namespace from hook

  #24716 opened Jun 28, 2022

• More typing and minor refactor for kubernetes

  #24719 opened Jun 28, 2022

• Added Support in ts_nodash

  #24724 opened Jun 29, 2022

• 24681: Fixed issue where cli.logs was giving character array instead of string

  #24726 opened Jun 29, 2022

31 Issues closed by 9 people

• `AirbyteHook` add cancel job option

  #24574 closed Jun 29, 2022

• Migration changes column to NOT NULL without updating NULL data first

  #24566 closed Jun 29, 2022

• Race condition between Triggerer and Scheduler

  #23824 closed Jun 28, 2022

• Add an AWS operator for Create RDS Database

  #23822 closed Jun 28, 2022

• Duplicate dag ids lead to confusing outcome rather than error.

  #24701 closed Jun 28, 2022

• Make port as optional argument

  #24022 closed Jun 28, 2022

• Scheduler gets permission error when running os.stat on dag file

  #24687 closed Jun 28, 2022

• Add SQL Provider

  #24422 closed Jun 28, 2022

• Partial of a KubernetesPodOperator does not allow for limit_cpu and limit_memory in the resources argument

  #23783 closed Jun 28, 2022

• TrinoHook connecting over https should support CertificateAuthentication from config extras

  #24693 closed Jun 28, 2022

• Mapped TriggerDagRunOperator causes SerializationError due to operator_extra_links 'property' object is not iterable

  #24653 closed Jun 28, 2022

• sql_alchemy_conn environment variable raises airflow.exceptions.AirflowConfigException

  #24690 closed Jun 27, 2022

• Snowflake Provider connection documentation is misleading

  #24572 closed Jun 27, 2022

• Add Templated SQL Check Operators for Columns and Tables

  #23874 closed Jun 27, 2022

• setting `resources` in `KubernetesPodOperator` breaks when used with `Operator.partial`

  #24669 closed Jun 27, 2022

• Migrate Snowflake system tests to new design

  #22434 closed Jun 26, 2022

• CI: Rewrite selective check script in Python

  #19971 closed Jun 25, 2022

• In my recent Airflow exploration, I'm trying to dump logs from Airflow to external systems, possibly without using FluentD, the target system is Splunk in my case.

  #24645 closed Jun 25, 2022

• Enable keys linked to TaskInstance map_index

  #24644 closed Jun 25, 2022

• Broken link example_external_task_marker_dag.html

  #23480 closed Jun 24, 2022

• ExternalTaskSensor should skip if soft_fail=True and external task in one of the failed_states

  #19754 closed Jun 24, 2022

• Airflow Grid view UI issues

  #24410 closed Jun 24, 2022

• Webserver shows wrong datetime (timezone) in log

  #23796 closed Jun 24, 2022

• All the task cells in the same row are showing same state (latest state)

  #24630 closed Jun 24, 2022

• Grid view disappears when hovering over skipped mapped tasks

  #24620 closed Jun 24, 2022

• db+ string in result backend but not metadata secret

  #18283 closed Jun 23, 2022

• Upgrade to FlaskAppBuilder 4.0.*

  #22397 closed Jun 22, 2022

• Status of testing Providers that were prepared on June 15, 2022

  #24471 closed Jun 22, 2022

• LocalExecutor run Async operators without a triggerer process

  #24602 closed Jun 22, 2022

• Plus sign in chart label should be replaced with an underscore

  #24604 closed Jun 22, 2022

• AWS Apache Airflow - Error! Please contact server admin.

  #24598 closed Jun 22, 2022

17 Issues opened by 16 people

• Celery Executor : After killing Redis or Airflow Worker Pod, queued Tasks not getting executed even after pod is up.

  #24731 opened Jun 29, 2022

• Google CloudRun job operator

  #24730 opened Jun 29, 2022

• BigQueryToBigQueryOperator failing with "Not found: Job ..." error

  #24729 opened Jun 29, 2022

• DAGs View: make Owner column hyperlink for a predefined URL

  #24728 opened Jun 29, 2022

• Trigger DAG from templated view tab producing bad request

  #24725 opened Jun 29, 2022

• Grid view - Show vertical lines at rounded date intervals

  #24723 opened Jun 29, 2022

• `default_wrap` setting for webserver is not working

  #24721 opened Jun 29, 2022

• Password autocompletion set off on login page

  #24700 opened Jun 28, 2022

• Clear one single mapped task instance

  #24699 opened Jun 28, 2022

• Error for Hive Server2 Connection Document

  #24692 opened Jun 28, 2022

• Docker is not pushing last line over xcom

  #24681 opened Jun 27, 2022

• Macro prev_execution_date is always empty

  #24678 opened Jun 27, 2022

• MySql SSL Mode

  #24675 opened Jun 27, 2022

• AirflowSecurityManager inconsistent with Flask-appbuilder when using @protect API in custom plugins

  #24674 opened Jun 27, 2022

• Erroneous comment about need to address AIRFLOW-1455 in airflow_local_settings.py

  #24642 opened Jun 24, 2022

• Redacted Variables override original value

  #24624 opened Jun 24, 2022

• Failed to retrieve data from Oracle database with UTF-8 charset

  #24618 opened Jun 23, 2022

85 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Unify DbApiHook.run() method with the methods which override it

  #23971 commented on Jun 28, 2022 • 11 new comments

• Rewrite the 'version' command to use Click

  #24591 commented on Jun 29, 2022 • 10 new comments

• Add Snowpark Operator that allows execution of Snowflake Snowpark code

  #24456 commented on Jun 26, 2022 • 9 new comments

• Serialize pod_override to JSON before pickling executor_config

  #24356 commented on Jun 25, 2022 • 6 new comments

• Move fallible ti.task.dag assignment back inside try/except block (#24533)

  #24592 commented on Jun 24, 2022 • 6 new comments

• After upgrade from Airflow 2.2.4, grid disappears for some DAGs

  #23588 commented on Jun 28, 2022 • 5 new comments

• Add parameter to turn off SQL query logging

  #24570 commented on Jun 25, 2022 • 5 new comments

• Scheduler crashes with psycopg2.errors.DeadlockDetected exception

  #23361 commented on Jun 28, 2022 • 4 new comments

• Make sure jobs triggered by airflow web are not identified as orphaned.

  #22976 commented on Jun 27, 2022 • 4 new comments

• perf(BigQuery): pass table_id as str type

  #23141 commented on Jun 25, 2022 • 4 new comments

• DAG regex flag in backfill command

  #23870 commented on Jun 29, 2022 • 4 new comments

• Create new databases from the ORM

  #24156 commented on Jun 28, 2022 • 4 new comments

• "Suggest a change on this page" button bug/change request

  #16654 commented on Jun 24, 2022 • 3 new comments

• KubernetesPodOperator/KubernetesExecutor: Failed to adopt pod 422

  #24015 commented on Jun 27, 2022 • 3 new comments

• Helmchart: Provision Standalone Dag Processor

  #23711 commented on Jun 27, 2022 • 3 new comments

• feat: add an operator to run a transfer job

  #24563 commented on Jun 23, 2022 • 3 new comments

• updated documentation for databricks operator

  #24599 commented on Jun 28, 2022 • 3 new comments

• Tasks intermittently gets terminated with SIGTERM on kubernetes executor

  #18041 commented on Jun 23, 2022 • 2 new comments

• Switch from 'cloudant' to 'ibmcloudant'

  #21004 commented on Jun 23, 2022 • 2 new comments

• Long running tasks being killed with CeleryKubernetesExecutor

  #24466 commented on Jun 24, 2022 • 2 new comments

• Improve ExternalTaskSensor execution date pattern

  #10883 commented on Jun 24, 2022 • 2 new comments

• Enable More PyDocStyle Checks

  #10742 commented on Jun 27, 2022 • 2 new comments

• Error when using connector in airflow snowflake hook: module 'snowflake' has no attribute 'connector'

  #24586 commented on Jun 27, 2022 • 2 new comments

• Max dagrun limitation should not stop failed dagrun from retry

  #23664 commented on Jun 28, 2022 • 2 new comments

• Add new operator for ClickHouse

  #10893 commented on Jun 29, 2022 • 2 new comments

• Improve ElasticsearchTaskHandler

  #21942 commented on Jun 29, 2022 • 2 new comments

• Add support for Delta Sharing protocol

  #22692 commented on Jun 27, 2022 • 2 new comments

• Introduce sla_miss metric

  #23402 commented on Jun 28, 2022 • 2 new comments

• Update dag-run.rst to ref dag_run.conf

  #24452 commented on Jun 22, 2022 • 2 new comments

• Add support for Salesforce bulk api

  #24473 commented on Jun 29, 2022 • 2 new comments

• add: container securityContext not available in podSecurityContext

  #24588 commented on Jun 29, 2022 • 2 new comments

• Datacatalog assets & system tests migration (AIP-47)

  #24600 commented on Jun 28, 2022 • 2 new comments

• Airflow Scheduler continuously restarts when using the CeleryKubernetesExecutor

  #24538 commented on Jun 22, 2022 • 1 new comment

• Task processes killed with WARNING - Recorded pid does not match the current pid

  #17507 commented on Jun 23, 2022 • 1 new comment

• TaskFlow AirflowSkipException causes downstream step to fail

  #24338 commented on Jun 23, 2022 • 1 new comment

• mini-scheduler raises AttributeError: 'NoneType' object has no attribute 'keys'

  #24525 commented on Jun 23, 2022 • 1 new comment

• `GoogleDriveToLocalOperator` gives conn_id error despite not taking conn_id as a param

  #24494 commented on Jun 23, 2022 • 1 new comment

• Triggering DAG from UI Creates 2 DAG runs if the DAG was paused

  #24534 commented on Jun 23, 2022 • 1 new comment

• Multi Schedulers don't synchronize task instance state when using Triggerer

  #20308 commented on Jun 25, 2022 • 1 new comment

• Rewrite Breeze in Python

  #12282 commented on Jun 25, 2022 • 1 new comment

• bigquery provider's - BigQueryCursor missing implementation for description property.

  #22328 commented on Jun 25, 2022 • 1 new comment

• Duplicate task instances starts if scheduler restarts in middle of execution of task

  #23520 commented on Jun 26, 2022 • 1 new comment

• In 'list users', clicking 'show record' button returns an error page

  #22035 commented on Jun 26, 2022 • 1 new comment

• Programmatic customization of run_id for scheduled DagRuns

  #21923 commented on Jun 26, 2022 • 1 new comment

• UI bug in recent chromium when search for a variable

  #22774 commented on Jun 27, 2022 • 1 new comment

• Add DAG "Skipped" status in case of "startDate" in the future while manually triggering a DAGRun

  #23911 commented on Jun 27, 2022 • 1 new comment

• Airflow requires to-be-deprecated flag when using MySQL

  #12562 commented on Jun 27, 2022 • 1 new comment

• Task Instances in the "shutdown" state prevent the scheduler from scheduling new tasks

  #14744 commented on Jun 28, 2022 • 1 new comment

• dag callbacks are missed sometimes

  #19767 commented on Jun 29, 2022 • 1 new comment

• Jinja templates should be rendered in dict keys

  #18938 commented on Jun 29, 2022 • 1 new comment

• Fix inconsitencies in checking edit permissions for a DAG

  #20346 commented on Jun 28, 2022 • 1 new comment

• Cloudera community provider added (Work in progress)

  #22659 commented on Jun 27, 2022 • 1 new comment

• Add impersonation_chain parameter for ComputeEngineSSHHook

  #22910 commented on Jun 25, 2022 • 1 new comment

• Document built in Timetables

  #23099 commented on Jun 29, 2022 • 1 new comment

• Kubernetes Pod Operator support running with multiple containers

  #23450 commented on Jun 23, 2022 • 1 new comment

• Add advanced secrets backend configurations

  #23560 commented on Jun 23, 2022 • 1 new comment

• Fix defaultAirflowRepository comment

  #23568 commented on Jun 26, 2022 • 1 new comment

• WIP - Enable kubernetes watch bookmarks. Handle 410 error in kubernetes_executor

  #23578 commented on Jun 25, 2022 • 1 new comment

• Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497)

  #23658 commented on Jun 22, 2022 • 1 new comment

• Add Tabular provider

  #23704 commented on Jun 27, 2022 • 1 new comment

• breeze test parallel

  #23715 commented on Jun 29, 2022 • 1 new comment

• Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection)

  #24079 commented on Jun 28, 2022 • 1 new comment

• Add support for KEDA HPA config to Helm Chart

  #24220 commented on Jun 28, 2022 • 1 new comment

• KubernetesExecutor: Use user-provided namespace from executor_config arg

  #24342 commented on Jun 25, 2022 • 1 new comment

• Fixed BigQueryCreateExternalTableOperator and its unit test (#24160)

  #24363 commented on Jun 29, 2022 • 1 new comment

• Add as_dict and custom query to BigQueryGetDataOperator

  #24468 commented on Jun 29, 2022 • 1 new comment

• Documentation on task mapping additions

  #24489 commented on Jun 28, 2022 • 1 new comment

• Add cache_ok flag to sqlalchemy TypeDecorators.

  #24499 commented on Jun 23, 2022 • 1 new comment

• Filter on XCOM key name when calculating map lengths in MappedOperator

  #24530 commented on Jun 28, 2022 • 1 new comment

• Error in docstring in the example DAG for Databricks example_databricks_sql.py

  #24597 commented on Jun 22, 2022 • 0 new comments

• Add support for creating backfills to the stable REST API

  #18816 commented on Jun 23, 2022 • 0 new comments

• Rewrite Airflow CLI commands with Click

  #22708 commented on Jun 23, 2022 • 0 new comments

• Currently we are working on ...

  #10176 commented on Jun 25, 2022 • 0 new comments

• DAGs with Dynamic Task Mapping are not rendered when cluster policies overwrite a parameter

  #24547 commented on Jun 29, 2022 • 0 new comments

• Add Airtable Hook

  #8589 commented on Jun 29, 2022 • 0 new comments

• Initial support for Mixpanel ETL applications

  #9738 commented on Jun 29, 2022 • 0 new comments

• Add DeltaSharing provider and sensor.

  #19473 commented on Jun 29, 2022 • 0 new comments

• Add Prometheus Provider

  #20053 commented on Jun 29, 2022 • 0 new comments

• Convert CLI commands to use Click

  #22613 commented on Jun 29, 2022 • 0 new comments

• Catch Fernet decode error in Connection

  #23105 commented on Jun 28, 2022 • 0 new comments

• Remove Tableau from Salesforce provider

  #23747 commented on Jun 26, 2022 • 0 new comments

• Don't refresh mapped instances list when states are final

  #23805 commented on Jun 28, 2022 • 0 new comments

• Add disable retry flag on backfill

  #23829 commented on Jun 27, 2022 • 0 new comments

• Send DAG timeout callbacks to processor outside of prohibit_commit

  #24366 commented on Jun 27, 2022 • 0 new comments

• Migrate Google example DAG mssql_to_gcs to new design AIP-47

  #24541 commented on Jun 24, 2022 • 0 new comments