Free Databricks Certified Data Engineer Professional Exam Databricks-Certified-Professional-Data-Engineer Exam Practice Test

A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

• Set the configuration delta.deduplicate = true.
• VACUUM the Delta table after each batch completes.
• Perform an insert-only merge with a matching condition on a unique key.
• Perform a full outer join on a unique key and overwrite existing data.
• Rely on Delta Lake schema enforcement to prevent duplicate records.

A Delta Lake table representing metadata about content posts from users has the following schema:user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATEThis table is partitioned by the date column. A query is run with the following filter: longitude < 20 & longitude > -20Which statement describes how data will be filtered?

• Statistics in the Delta Log will be used to identify partitions that might Include files in the filteredrange.
• No file skipping will occur because the optimizer does not know the relationship between the partition column and the longitude.
• The Delta Engine will use row-level statistics in the transaction log to identify the flies that meet the filter criteria.
• Statistics in the Delta Log will be used to identify data files that might include records in the filtered range.
• The Delta Engine will scan the parquet file footers to identify each row that meets the filter criteria.

What statement is true regarding the retention of job run history?

• It is retained until you export or delete job run logs
• It is retained for 30 days, during which time you can deliver job run logs to DBFS or S3
• t is retained for 60 days, during which you can export notebook run results to HTML
• It is retained for 60 days, after which logs are archived
• It is retained for 90 days or until the run-id is re-used through custom run configuration

Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?

• spark.sql.files.maxPartitionBytes
• spark.sql.autoBroadcastJoinThreshold
• spark.sql.files.openCostInBytes
• spark.sql.adaptive.coalescePartitions.minPartitionNum
• spark.sql.adaptive.advisoryPartitionSizeInBytes

A production cluster has 3 executor nodes and uses the same virtual machine type for the driver and executor.When evaluating the Ganglia Metrics for this cluster, which indicator would signal a bottleneck caused by code executing on the driver?

• The five Minute Load Average remains consistent/flat
• Bytes Received never exceeds 80 million bytes per second
• Total Disk Space remains constant
• Network I/O never spikes
• Overall cluster CPU utilization is around 25%

A CHECK constraint has been successfully added to the Delta table named activity_details using the following logic:A batch job is attempting to insert new records to the table, including a record where latitude = 45.50 and longitude = 212.67.Which statement describes the outcome of this batch insert?

• The write will fail when the violating record is reached; any records previously processed will be recorded to the target table.
• The write will fail completely because of the constraint violation and no records will be inserted into the target table.
• The write will insert all records except those that violate the table constraints; the violating records will be recorded to a quarantine table.
• The write will include all records in the target table; any violations will be indicated in the boolean column named valid_coordinates.
• The write will insert all records except those that violate the table constraints; the violating records will be reported in a warning log.

A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.Which situation is causing increased duration of the overall job?

• Task queueing resulting from improper thread pool assignment.
• Spill resulting from attached volume storage being too small.
• Network latency due to some cluster nodes being in different regions from the source data
• Skew caused by more data being assigned to a subset of spark-partitions.
• Credential validation errors while pulling data from an external system.

Which is a key benefit of an end-to-end test?

• It closely simulates real world usage of your application.
• It pinpoint errors in the building blocks of your application.
• It provides testing coverage for all code paths and branches.
• It makes it easier to automate your test suite

Review the following error traceback:Which statement describes the error being raised?

• The code executed was PvSoark but was executed in a Scala notebook.
• There is no column in the table named heartrateheartrateheartrate
• There is a type error because a column object cannot be multiplied.
• There is a type error because a DataFrame object cannot be multiplied.
• There is a syntax error because the heartrate column is not correctly identified as a column.

Which statement describes Delta Lake optimized writes?

• A shufle occurs prior to writing to try to group data together resulting in fewer files instead of each executor writing multiple files based on directory partitions.
• Optimized writes logical partitions instead of directory partitions partition boundaries are only represented in metadata fewer small files are written.
• An asynchronous job runs after the write completes to detect if files could be further compacted; yes, an OPTIMIZE job is executed toward a default of 1 GB.
• Before a job cluster terminates, OPTIMIZE is executed on all tables modified during the most recent job.