Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
B
bcom-tp-etl-transformation-pipelines
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
general
bcom-tp-etl-transformation-pipelines
Commits
e89bb096
Commit
e89bb096
authored
Aug 01, 2023
by
Cristian Aguirre
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Update 31-07-23. Update README
parent
ab36e03a
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
1 deletion
+5
-1
Generation.py
dags/components/Generation.py
+5
-1
No files found.
dags/components/Generation.py
View file @
e89bb096
...
...
@@ -8,6 +8,7 @@ from airflow.decorators import task
from
airflow.exceptions
import
AirflowSkipException
from
enums.ProcessStatusEnum
import
ProcessStatusEnum
from
enums.DatabaseTypeEnum
import
DatabaseTypeEnum
from
components.S3Route
import
save_df_to_s3
,
load_obj_to_s3
from
components.Utils
import
select_multiple
,
create_temp_file
,
delete_temp_dir
from
components.Control
import
get_tasks_from_control
,
update_new_process
...
...
@@ -90,8 +91,11 @@ def generate_and_deploy(command: str, intern_conn, params: Dict[str, Any], timez
delimiter
=
params
[
"delimiter"
]
tmp_path
=
params
[
"tmp_path"
]
tmp_file
=
create_temp_file
(
tmp_path
,
filename_mask
,
file_type
,
tablename
,
timezone
,
pattern
)
logger
.
info
(
tmp_file
)
logger
.
info
(
f
"Ruta creada: {tmp_file}"
)
logger
.
debug
(
f
"TABLA: {tablename}"
)
steps
=
get_steps
(
tablename
,
chunksize
,
engine
,
True
)
if
intern_conn
.
db_type
==
DatabaseTypeEnum
.
ORACLE
.
value
:
tablename
=
f
"SELECT * FROM {tablename}"
iterator
=
get_iterator
(
tablename
,
chunksize
,
engine
)
logger
.
info
(
f
"Total de pasos para generar archivo resultado: {steps}"
)
for
step
in
range
(
steps
):
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment