Aug 04 22:21:02.330184 ip-10-0-142-146 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 22:21:02.763094 ip-10-0-142-146 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 22:21:02.763094 ip-10-0-142-146 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 22:21:02.763094 ip-10-0-142-146 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 22:21:02.763094 ip-10-0-142-146 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 22:21:02.763094 ip-10-0-142-146 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 22:21:02.763720 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.763157    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 22:21:02.768205 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768189    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.768205 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768203    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768209    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768213    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768217    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768220    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768223    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768226    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768228    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768232    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768235    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768238    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768240    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768243    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768245    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768248    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768250    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768253    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768256    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768258    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.768267 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768261    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768263    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768266    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768269    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768272    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768274    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768277    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768280    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768288    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768290    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768293    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768296    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768299    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768301    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768304    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768306    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768309    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768311    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768313    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.768721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768316    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768318    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768321    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768323    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768326    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768329    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768331    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768334    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768337    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768340    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768342    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768344    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768347    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768349    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768354    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768358    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768361    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768363    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768366    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768368    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.769207 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768371    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768373    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768376    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768379    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768381    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768383    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768386    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768388    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768392    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768395    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768398    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768401    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768404    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768407    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768410    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768412    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768415    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768418    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768421    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.769700 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768424    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768426    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768429    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768431    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768434    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768436    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768439    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768441    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768444    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768447    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768449    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768452    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768454    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768456    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768459    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768462    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768464    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768467    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768469    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768472    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.770175 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.768474    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769157    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769164    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769166    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769169    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769172    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769174    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769178    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769180    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769183    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769186    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769189    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769192    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769195    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769197    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769199    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769202    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769205    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769207    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769210    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.770681 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769213    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769215    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769218    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769220    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769223    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769226    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769228    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769231    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769233    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769236    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769238    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769240    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769243    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769245    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769248    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769250    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769252    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769255    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769257    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769260    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.771191 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769263    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769267    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769270    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769273    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769275    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769278    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769280    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769283    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769286    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769288    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769292    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769296    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769301    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769304    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769307    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769310    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769313    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769316    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769318    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.771683 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769321    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769324    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769326    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769329    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769331    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769335    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769338    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769341    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769344    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769347    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769350    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769352    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769355    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769359    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769361    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769364    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769366    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769369    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769371    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.772239 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769374    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769376    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769387    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769390    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769392    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769395    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769398    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769401    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769404    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769406    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769409    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769411    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769414    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769417    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769419    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769422    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769424    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769427    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769429    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769432    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769434    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.772698 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769437    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769798    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769803    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769805    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769808    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769811    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769813    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769816    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769818    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769821    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769823    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769826    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769829    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769831    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769834    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769836    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769839    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769842    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769844    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769847    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.773237 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769849    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769852    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769854    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769857    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769859    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769861    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769864    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769866    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769869    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769871    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769874    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769876    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769879    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769881    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769884    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769886    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769888    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769906    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769909    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.773725 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769912    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769914    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769917    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769919    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769922    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769924    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769927    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769929    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769931    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769934    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769937    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769939    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769942    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769944    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769947    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769950    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769952    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769955    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769959    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769962    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.774213 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769965    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769968    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769970    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769973    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769976    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769979    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769981    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769984    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769986    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769989    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769992    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769994    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.769998    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770001    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770004    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770007    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770009    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770012    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770016    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770018    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.774714 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770020    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770023    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770026    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770028    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770030    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770033    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770035    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770038    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770040    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770043    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770045    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770049    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770052    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770054    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770057    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770060    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770063    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770066    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770068    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770071    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.775215 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770073    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770151    2582 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770161    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770171    2582 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770178    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770184    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770189    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770196    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770201    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770204    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770207    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770212    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770215    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770218    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770221    2582 flags.go:64] FLAG: --cgroup-root=""
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770224    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770227    2582 flags.go:64] FLAG: --client-ca-file=""
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770230    2582 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770233    2582 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770237    2582 flags.go:64] FLAG: --cluster-domain=""
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770239    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770242    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770245    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770248    2582 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 22:21:02.775730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770251    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770254    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770257    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770260    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770263    2582 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770266    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770271    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770274    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770277    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770282    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770285    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770288    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770290    2582 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770293    2582 flags.go:64] FLAG: --enable-server="true"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770296    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770300    2582 flags.go:64] FLAG: --event-burst="100"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770303    2582 flags.go:64] FLAG: --event-qps="50"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770306    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770309    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770312    2582 flags.go:64] FLAG: --eviction-hard=""
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770316    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770319    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770321    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770324    2582 flags.go:64] FLAG: --eviction-soft=""
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770327    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 22:21:02.776360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770330    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770332    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770335    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770338    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770340    2582 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770343    2582 flags.go:64] FLAG: --feature-gates=""
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770347    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770350    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770353    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770356    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770359    2582 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770361    2582 flags.go:64] FLAG: --help="false"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770365    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770370    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770373    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770375    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770379    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770382    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770384    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770387    2582 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770390    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770392    2582 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770395    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770398    2582 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 22:21:02.776997 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770401    2582 flags.go:64] FLAG: --kube-reserved=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770403    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770406    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770409    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770412    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770415    2582 flags.go:64] FLAG: --lock-file=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770418    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770420    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770423    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770428    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770431    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770433    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770436    2582 flags.go:64] FLAG: --logging-format="text"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770439    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770442    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770444    2582 flags.go:64] FLAG: --manifest-url=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770447    2582 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770451    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770454    2582 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770458    2582 flags.go:64] FLAG: --max-pods="110"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770462    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770466    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770469    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770472    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770475    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 22:21:02.777588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770478    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770480    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770487    2582 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770490    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770493    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770496    2582 flags.go:64] FLAG: --pod-cidr=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770498    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770503    2582 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770506    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770509    2582 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770512    2582 flags.go:64] FLAG: --port="10250"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770515    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770518    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-058de74e00af1c94e"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770521    2582 flags.go:64] FLAG: --qos-reserved=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770525    2582 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770528    2582 flags.go:64] FLAG: --register-node="true"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770530    2582 flags.go:64] FLAG: --register-with-taints=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770534    2582 flags.go:64] FLAG: --registry-burst="10"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770537    2582 flags.go:64] FLAG: --registry-qps="5"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770539    2582 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770542    2582 flags.go:64] FLAG: --reserved-memory=""
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770545    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770548    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770551    2582 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770553    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 22:21:02.778211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770556    2582 flags.go:64] FLAG: --runonce="false"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770558    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770561    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770569    2582 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770572    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770574    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770577    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770580    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770583    2582 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770586    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770589    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770591    2582 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770594    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770597    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770600    2582 flags.go:64] FLAG: --system-cgroups=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770603    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770607    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770610    2582 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770613    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770616    2582 flags.go:64] FLAG: --tls-min-version=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770620    2582 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770622    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770625    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770628    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770630    2582 flags.go:64] FLAG: --v="2"
Aug 04 22:21:02.778815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770634    2582 flags.go:64] FLAG: --version="false"
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770638    2582 flags.go:64] FLAG: --vmodule=""
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770641    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.770644    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770760    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770764    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770766    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770769    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770772    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770774    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770780    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770783    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770785    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770789    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770791    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770794    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770796    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770799    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770801    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770804    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.779493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770807    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770809    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770812    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770814    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770816    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770819    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770821    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770824    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770827    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770829    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770832    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770834    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770837    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770839    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770842    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770846    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770849    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770852    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770854    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.780014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770857    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770860    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770862    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770868    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770871    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770873    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770876    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770878    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770881    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770884    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770886    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770889    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770908    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770911    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770914    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770916    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770919    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770922    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770924    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.780502 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770927    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770929    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770931    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770939    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770942    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770944    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770947    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770949    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770951    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770955    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770959    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770962    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770965    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770967    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770970    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770973    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770980    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770984    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770987    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.781003 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770990    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770992    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770995    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.770998    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771001    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771003    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771006    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771009    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771011    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771014    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771017    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771019    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771022    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771024    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771027    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771030    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771033    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771040    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771043    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771046    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.781490 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771048    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771051    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771053    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771056    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771058    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.771061    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.771066    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.777612    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.777628    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.777643    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.777650    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777704    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777709    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777712    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777715    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.782046 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777718    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777721    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777723    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777726    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777729    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777731    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777734    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777737    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777740    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777742    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777745    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777747    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777749    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777752    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777754    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777757    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777759    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777762    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777764    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777766    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777769    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.782431 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777771    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777774    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777776    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777779    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777783    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777786    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777789    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777792    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777794    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777797    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777799    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777802    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777805    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777807    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777810    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777812    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777815    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777817    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777820    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.783013 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777822    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777826    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777830    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777834    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777836    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777839    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777842    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777844    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777847    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777849    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777852    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777854    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777857    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777859    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777862    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777865    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777867    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777871    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777877    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.783498 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777879    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777882    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777884    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777886    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777889    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777909    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777912    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777914    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777917    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777919    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777922    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777924    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777926    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777929    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777931    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777934    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777936    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777939    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777941    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777943    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.783989 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777946    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777948    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777952    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777955    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777958    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777961    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777964    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777967    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777970    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777972    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777976    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777980    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777984    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777986    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777989    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.784482 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.777991    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.777996    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778106    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778111    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778114    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778116    2582 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778119    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778122    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778124    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778127    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778129    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778131    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778134    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778136    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778139    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778141    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 22:21:02.784854 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778143    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778146    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778148    2582 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778150    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778153    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778155    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778157    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778160    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778162    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778165    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778168    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778170    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778173    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778175    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778178    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778180    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778183    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778185    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778187    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778189    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 22:21:02.785274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778192    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778194    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778197    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778199    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778201    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778204    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778206    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778209    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778211    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778214    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778217    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778219    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778222    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778224    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778228    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778231    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778235    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778237    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778240    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 22:21:02.785789 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778243    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778245    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778248    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778250    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778254    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778256    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778259    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778262    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778264    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778267    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778269    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778271    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778274    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778277    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778280    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778283    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778285    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778288    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778290    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778292    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 22:21:02.786274 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778295    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778297    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778300    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778302    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778305    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778307    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778309    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778312    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778314    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778317    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778319    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778321    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778324    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778326    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778328    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778331    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778333    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778336    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778338    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 22:21:02.786813 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778341    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778343    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778346    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778348    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778351    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778355    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:02.778358    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.778363    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.778453    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.780409    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.781748    2582 server.go:1019] "Starting client certificate rotation"
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.781850    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.781914    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 22:21:02.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.784310    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 22:21:02.787642 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.786644    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 22:21:02.798513 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.798492    2582 log.go:25] "Validated CRI v1 runtime API"
Aug 04 22:21:02.800519 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.800504    2582 log.go:25] "Validated CRI v1 image API"
Aug 04 22:21:02.801351 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.801336    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 22:21:02.805565 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.805535    2582 fs.go:135] Filesystem UUIDs: map[71830724-7a12-4bb4-b5fc-bfb43dc74277:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f2fbd7c0-1b0e-445b-8a0e-a9a11e521ac2:/dev/nvme0n1p4]
Aug 04 22:21:02.805631 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.805562    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 22:21:02.808237 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.808216    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 22:21:02.811965 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.811844    2582 manager.go:216] Machine: {Timestamp:2026-08-04 22:21:02.809519252 +0000 UTC m=+0.351012418 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127575 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b5608e62f3177f0a977f0f196284 SystemUUID:ec22b560-8e62-f317-7f0a-977f0f196284 BootID:b7bb8295-5fbe-48cf-9c31-4883d4e01554 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:eb:df:64:9e:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:eb:df:64:9e:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:09:6d:b9:d3:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 22:21:02.811965 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.811962    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 22:21:02.812071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.812034    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 22:21:02.815133 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.815113    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 22:21:02.815269 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.815137    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 22:21:02.815853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.815841    2582 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 22:21:02.815880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.815856    2582 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 22:21:02.815880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.815870    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 22:21:02.816580 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.816570    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 22:21:02.817305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.817296    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 22:21:02.820433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820422    2582 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 22:21:02.820535 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820526    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 22:21:02.820574 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820557    2582 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 22:21:02.820574 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820567    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 22:21:02.820639 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820581    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 22:21:02.820639 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820590    2582 kubelet.go:398] "Adding apiserver pod source"
Aug 04 22:21:02.820639 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820598    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 22:21:02.820939 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820924    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 22:21:02.821000 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.820943    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 22:21:02.824401 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.824384    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 22:21:02.826154 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.826135    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 22:21:02.826210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.826170    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 22:21:02.826907 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.826877    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kh8d"
Aug 04 22:21:02.827571 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827557    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 22:21:02.827571 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827574    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827580    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827586    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827600    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827608    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827619    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827624    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827629    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827635    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827642    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 22:21:02.827689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.827647    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 22:21:02.828416 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.828406    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 22:21:02.828416 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.828415    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 22:21:02.831115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.831100    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 22:21:02.831217 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.831197    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 22:21:02.831257 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.831200    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 22:21:02.831696 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.831684    2582 server.go:1268] "Started kubelet"
Aug 04 22:21:02.831800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.831744    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 22:21:02.831925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.831776    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 22:21:02.832004 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.831942    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 22:21:02.832533 ip-10-0-142-146 systemd[1]: Started Kubernetes Kubelet.
Aug 04 22:21:02.835491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.835471    2582 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 22:21:02.835666 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.835650    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 22:21:02.837061 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.837041    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kh8d"
Aug 04 22:21:02.839029 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.839008    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 22:21:02.840356 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.839340    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-146.ec2.internal.18c8ba462dd2a202  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-146.ec2.internal,UID:ip-10-0-142-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-146.ec2.internal,},FirstTimestamp:2026-08-04 22:21:02.831657474 +0000 UTC m=+0.373150638,LastTimestamp:2026-08-04 22:21:02.831657474 +0000 UTC m=+0.373150638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-146.ec2.internal,}"
Aug 04 22:21:02.840497 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.840479    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 22:21:02.840497 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.840487    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 22:21:02.840630 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.840562    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:02.841091 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841057    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 22:21:02.841234 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841203    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 22:21:02.841234 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841223    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 22:21:02.841356 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841297    2582 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 22:21:02.841356 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841306    2582 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 22:21:02.841471 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841461    2582 factory.go:153] Registering CRI-O factory
Aug 04 22:21:02.841525 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841507    2582 factory.go:223] Registration of the crio container factory successfully
Aug 04 22:21:02.841585 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841556    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 22:21:02.841585 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841568    2582 factory.go:55] Registering systemd factory
Aug 04 22:21:02.841585 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841576    2582 factory.go:223] Registration of the systemd container factory successfully
Aug 04 22:21:02.841724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841603    2582 factory.go:103] Registering Raw factory
Aug 04 22:21:02.841724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.841617    2582 manager.go:1196] Started watching for new ooms in manager
Aug 04 22:21:02.842017 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.842003    2582 manager.go:349] Starting recovery of all containers
Aug 04 22:21:02.847675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.847656    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:02.850301 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.850281    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-146.ec2.internal\" not found" node="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:02.852191 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.852177    2582 manager.go:354] Recovery completed
Aug 04 22:21:02.856439 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.856423    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:02.858829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.858813    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:02.858881 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.858855    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:02.858881 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.858870    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:02.859408 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.859393    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 22:21:02.859408 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.859406    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 22:21:02.859517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.859431    2582 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 22:21:02.859517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.859409    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:02.860596 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860582    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:02.860656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860607    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:02.860656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860616    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:02.860656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860651    2582 policy_none.go:49] "None policy: Start"
Aug 04 22:21:02.860749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860665    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 22:21:02.860749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.860675    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 22:21:02.861587 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.861575    2582 policy_none.go:47] "Start"
Aug 04 22:21:02.905539 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905407    2582 manager.go:337] "Starting Device Plugin manager"
Aug 04 22:21:02.905615 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.905555    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 22:21:02.905615 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905565    2582 server.go:87] "Starting device plugin registration server"
Aug 04 22:21:02.905785 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905774    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 22:21:02.905817 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905787    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 22:21:02.906010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905908    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 22:21:02.906010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905974    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 22:21:02.906010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:02.905982    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 22:21:02.906469 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.906448    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 22:21:02.906513 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:02.906494    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.002670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.002641    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 22:21:03.005045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.003922    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 22:21:03.005045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.003965    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 22:21:03.005045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.003987    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 22:21:03.005045 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.004024    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 22:21:03.006037 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006016    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:03.006155 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006054    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:03.006865 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006848    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:03.006974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006879    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:03.006974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006911    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:03.006974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.006938    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.012230 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.012216    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.012300 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.012237    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-146.ec2.internal\": node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.026106 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.026050    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.104262 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.104230    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"]
Aug 04 22:21:03.104352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.104307    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:03.105144 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.105129    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:03.105219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.105157    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:03.105219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.105166    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:03.106499 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.106486    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:03.106674 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.106661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.106742 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.106694    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:03.107153 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107139    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:03.107221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107164    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:03.107221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107173    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:03.107221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107175    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:03.107221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107196    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:03.107221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.107205    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:03.108476 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.108460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.108531 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.108495    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 22:21:03.109165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.109147    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 22:21:03.109272 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.109177    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 22:21:03.109272 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.109187    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeHasSufficientPID"
Aug 04 22:21:03.123457 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.123439    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-146.ec2.internal\" not found" node="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.126133 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.126109    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.127635 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.127617    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-146.ec2.internal\" not found" node="ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.144236 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.144215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.144315 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.144239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.144315 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.144257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b56d4e9c7e28a671fb29fb60ed5ddf2b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-146.ec2.internal\" (UID: \"b56d4e9c7e28a671fb29fb60ed5ddf2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.226875 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.226857    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.245235 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.245286 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.245337 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.245337 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0ac1cd241ea2e991901d610263cbe9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal\" (UID: \"2e0ac1cd241ea2e991901d610263cbe9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.245444 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b56d4e9c7e28a671fb29fb60ed5ddf2b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-146.ec2.internal\" (UID: \"b56d4e9c7e28a671fb29fb60ed5ddf2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.245444 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.245400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b56d4e9c7e28a671fb29fb60ed5ddf2b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-146.ec2.internal\" (UID: \"b56d4e9c7e28a671fb29fb60ed5ddf2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.327597 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.327545    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.430781 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.430758    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.430781 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.430768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.430893 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.430793    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.535126 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.535089    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.636714 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.636630    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.738097 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.738055    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.782866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.782828    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 22:21:03.783327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.783002    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 22:21:03.783327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.783036    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 22:21:03.838546 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:03.838522    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-146.ec2.internal\" not found"
Aug 04 22:21:03.839716 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.839687    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 22:16:03 +0000 UTC" deadline="2028-02-21 11:37:19.086567317 +0000 UTC"
Aug 04 22:21:03.839769 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.839718    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13573h16m15.246852942s"
Aug 04 22:21:03.857173 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.857151    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:03.858985 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.858962    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:03.941501 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.941474    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.953864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.953042    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 22:21:03.954288 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.954267    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal"
Aug 04 22:21:03.963471 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:03.963451    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 22:21:04.008014 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:04.007985    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56d4e9c7e28a671fb29fb60ed5ddf2b.slice/crio-5d80624fcc571efbab3e9357aa15557c27ea1681cb63e8f03daa487c1b5ba298 WatchSource:0}: Error finding container 5d80624fcc571efbab3e9357aa15557c27ea1681cb63e8f03daa487c1b5ba298: Status 404 returned error can't find the container with id 5d80624fcc571efbab3e9357aa15557c27ea1681cb63e8f03daa487c1b5ba298
Aug 04 22:21:04.008724 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:04.008707    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0ac1cd241ea2e991901d610263cbe9.slice/crio-0fae901e81c0d4b52442c24c3aa6efe9adc2d8ead676b47553589441099fbc14 WatchSource:0}: Error finding container 0fae901e81c0d4b52442c24c3aa6efe9adc2d8ead676b47553589441099fbc14: Status 404 returned error can't find the container with id 0fae901e81c0d4b52442c24c3aa6efe9adc2d8ead676b47553589441099fbc14
Aug 04 22:21:04.013647 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.013619    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 22:21:04.824032 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.823998    2582 apiserver.go:52] "Watching apiserver"
Aug 04 22:21:04.831880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.831852    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 22:21:04.833966 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.833930    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-lfxnl","kube-system/global-pull-secret-syncer-5855j","kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal","openshift-image-registry/node-ca-4z68c","openshift-multus/multus-additional-cni-plugins-89svx","openshift-multus/multus-stpwt","openshift-network-diagnostics/network-check-target-2wxcj","openshift-network-operator/iptables-alerter-fl6pd","kube-system/konnectivity-agent-ch8lh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf","openshift-cluster-node-tuning-operator/tuned-lmhfd","openshift-dns/node-resolver-tsjtt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal","openshift-multus/network-metrics-daemon-bqcgb"]
Aug 04 22:21:04.836184 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.836156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:04.836283 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.836255    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:04.837223 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.837201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.839511 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.839488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.839608 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.839561    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:04.839800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.839776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fbld5\""
Aug 04 22:21:04.840098 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.840078    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.840098 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.840092    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.841094 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.841072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.841400 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.841376    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 22:21:04.843224 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.842707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.843550 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.843355    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 22:21:04.843550 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.843421    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.843550 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.843432    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4lqvx\""
Aug 04 22:21:04.843550 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.843545    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.844621 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.844262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.844904 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.844883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 22:21:04.845258 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.845241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 22:21:04.845512 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.845491    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.845703 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.845685    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.845892 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.845871    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dprxj\""
Aug 04 22:21:04.846241 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.846078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:04.846241 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.846133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 22:21:04.846542 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.846525    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s9jb9\""
Aug 04 22:21:04.846652 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.846623    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 22:21:04.847328 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.847306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:04.848485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.848317    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6z66p\""
Aug 04 22:21:04.848485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.848329    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 22:21:04.848485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.848385    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 22:21:04.848718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.848632    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:04.848718 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.848694    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:04.849220 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.849200    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.849510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.849495    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 22:21:04.849609 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.849531    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.849871 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.849801    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r5scw\""
Aug 04 22:21:04.850605 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.850191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.851802 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.851785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.852537 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-system-cni-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.852617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 22:21:04.852683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852645    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.852683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852656    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pkcxp\""
Aug 04 22:21:04.852683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852662    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-modprobe-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-sys\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-os-release\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-binary-copy\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.852846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-tmp\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysconfig\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-var-lib-kubelet\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.852978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhf6\" (UniqueName: \"kubernetes.io/projected/0321c2f7-349d-41ee-844b-7710cdab0e8c-kube-api-access-pzhf6\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-kubernetes\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-lib-modules\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-host\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjb7x\" (UniqueName: \"kubernetes.io/projected/03c5ee73-fd6c-4f21-a662-ec218702b7d9-kube-api-access-tjb7x\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9nw\" (UniqueName: \"kubernetes.io/projected/a38d7069-0118-40fd-ba16-aaffbb74a00c-kube-api-access-lz9nw\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-systemd\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-run\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fklc\" (UniqueName: \"kubernetes.io/projected/604304b6-6b2a-4511-af73-402e56662794-kube-api-access-5fklc\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0321c2f7-349d-41ee-844b-7710cdab0e8c-serviceca\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-cnibin\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-conf\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-etc-tuned\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.853898 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.853505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0321c2f7-349d-41ee-844b-7710cdab0e8c-host\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.854258 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854069    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 22:21:04.854485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854463    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.854634 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 22:21:04.854825 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854809    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 22:21:04.854908 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:04.855002 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.854952    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.855118 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.855101    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mbhs2\""
Aug 04 22:21:04.855232 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.855215    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 22:21:04.857228 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.857208    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 22:21:04.857460 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.857419    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 22:21:04.857811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.857794    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qv7t\""
Aug 04 22:21:04.859829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.859799    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 22:21:04.877496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.877474    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mbfs"
Aug 04 22:21:04.886394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.886373    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9mbfs"
Aug 04 22:21:04.933881 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.933854    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:04.942798 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.942778    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-tmp\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qvq\" (UniqueName: \"kubernetes.io/projected/baf26828-2a89-47d0-abed-a969ada126d1-kube-api-access-98qvq\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-netd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkrk\" (UniqueName: \"kubernetes.io/projected/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-kube-api-access-mtkrk\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysconfig\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-socket-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjrq\" (UniqueName: \"kubernetes.io/projected/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-kube-api-access-znjrq\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-kubernetes\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-k8s-cni-cncf-io\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-iptables-alerter-script\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-slash\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-ovn\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-node-log\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9nw\" (UniqueName: \"kubernetes.io/projected/a38d7069-0118-40fd-ba16-aaffbb74a00c-kube-api-access-lz9nw\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.955913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-multus\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-agent-certs\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-tmp-dir\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-systemd\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fklc\" (UniqueName: \"kubernetes.io/projected/604304b6-6b2a-4511-af73-402e56662794-kube-api-access-5fklc\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0321c2f7-349d-41ee-844b-7710cdab0e8c-serviceca\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-bin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-multus-certs\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-script-lib\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-hosts-file\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-conf\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-etc-tuned\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0321c2f7-349d-41ee-844b-7710cdab0e8c-host\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.954986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:04.956800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-kubelet\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-host-slash\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955065    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-multus-daemon-config\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysconfig\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-conf\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955350    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-kubernetes\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-device-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-sys-fs\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-kubelet\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-system-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-systemd\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0321c2f7-349d-41ee-844b-7710cdab0e8c-host\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.957601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-os-release\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-var-lib-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-var-lib-kubelet\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhf6\" (UniqueName: \"kubernetes.io/projected/0321c2f7-349d-41ee-844b-7710cdab0e8c-kube-api-access-pzhf6\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-sysctl-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.955986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-konnectivity-ca\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-var-lib-kubelet\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcs9h\" (UniqueName: \"kubernetes.io/projected/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-kube-api-access-fcs9h\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-dbus\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0321c2f7-349d-41ee-844b-7710cdab0e8c-serviceca\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-env-overrides\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-lib-modules\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-hostroot\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-etc-kubernetes\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.958718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-netns\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-lib-modules\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-etc-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-config\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-host\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjb7x\" (UniqueName: \"kubernetes.io/projected/03c5ee73-fd6c-4f21-a662-ec218702b7d9-kube-api-access-tjb7x\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-host\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-netns\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-kubelet-config\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-systemd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-log-socket\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.956967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-run\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-cnibin\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-conf-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-cnibin\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.959558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-systemd-units\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-socket-dir-parent\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-system-cni-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-cnibin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-system-cni-dir\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pbz\" (UniqueName: \"kubernetes.io/projected/a449619b-0633-4b78-9196-cd078b370979-kube-api-access-l6pbz\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-run\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-registration-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-modprobe-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-sys\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-os-release\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.957410    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.957492    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:05.457452988 +0000 UTC m=+2.998946162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.960451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-binary-copy\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-cni-binary-copy\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-bin\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-etc-modprobe-d\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a38d7069-0118-40fd-ba16-aaffbb74a00c-cni-binary-copy\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/604304b6-6b2a-4511-af73-402e56662794-sys\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.961074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.957832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a38d7069-0118-40fd-ba16-aaffbb74a00c-os-release\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.961807 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.961782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-tmp\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.965105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.965081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhf6\" (UniqueName: \"kubernetes.io/projected/0321c2f7-349d-41ee-844b-7710cdab0e8c-kube-api-access-pzhf6\") pod \"node-ca-4z68c\" (UID: \"0321c2f7-349d-41ee-844b-7710cdab0e8c\") " pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:04.965374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.965355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/604304b6-6b2a-4511-af73-402e56662794-etc-tuned\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.966004 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.965983    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:04.966104 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.966012    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:04.966104 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.966037    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:04.966104 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:04.966099    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:05.46608022 +0000 UTC m=+3.007573389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:04.968596 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.968552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjb7x\" (UniqueName: \"kubernetes.io/projected/03c5ee73-fd6c-4f21-a662-ec218702b7d9-kube-api-access-tjb7x\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:04.969695 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.969675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fklc\" (UniqueName: \"kubernetes.io/projected/604304b6-6b2a-4511-af73-402e56662794-kube-api-access-5fklc\") pod \"tuned-lmhfd\" (UID: \"604304b6-6b2a-4511-af73-402e56662794\") " pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:04.970114 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.970091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9nw\" (UniqueName: \"kubernetes.io/projected/a38d7069-0118-40fd-ba16-aaffbb74a00c-kube-api-access-lz9nw\") pod \"multus-additional-cni-plugins-89svx\" (UID: \"a38d7069-0118-40fd-ba16-aaffbb74a00c\") " pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:04.981647 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:04.981624    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 22:21:05.010602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.010551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal" event={"ID":"b56d4e9c7e28a671fb29fb60ed5ddf2b","Type":"ContainerStarted","Data":"5d80624fcc571efbab3e9357aa15557c27ea1681cb63e8f03daa487c1b5ba298"}
Aug 04 22:21:05.012053 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.012029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal" event={"ID":"2e0ac1cd241ea2e991901d610263cbe9","Type":"ContainerStarted","Data":"0fae901e81c0d4b52442c24c3aa6efe9adc2d8ead676b47553589441099fbc14"}
Aug 04 22:21:05.058213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-kubelet\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.058336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-host-slash\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.058336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-multus-daemon-config\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.058336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-device-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.058336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-sys-fs\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.058336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-kubelet\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-system-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-os-release\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-var-lib-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-konnectivity-ca\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcs9h\" (UniqueName: \"kubernetes.io/projected/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-kube-api-access-fcs9h\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-dbus\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.058608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-env-overrides\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-hostroot\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-etc-kubernetes\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-netns\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-etc-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-config\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-netns\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-kubelet-config\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-systemd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-log-socket\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-conf-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-systemd-units\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.058940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-socket-dir-parent\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-cnibin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pbz\" (UniqueName: \"kubernetes.io/projected/a449619b-0633-4b78-9196-cd078b370979-kube-api-access-l6pbz\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-registration-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-system-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-netns\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-kubelet\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-etc-kubernetes\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059151    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-device-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-etc-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-cni-binary-copy\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-kubelet\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-netns\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-cni-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-kubelet-config\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-hostroot\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-registration-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-dbus\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-socket-dir-parent\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-host-slash\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.060033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-log-socket\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-cnibin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-multus-conf-dir\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-os-release\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-bin\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-bin\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-var-lib-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qvq\" (UniqueName: \"kubernetes.io/projected/baf26828-2a89-47d0-abed-a969ada126d1-kube-api-access-98qvq\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-sys-fs\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-openvswitch\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-systemd-units\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-env-overrides\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-netd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-systemd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.060836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkrk\" (UniqueName: \"kubernetes.io/projected/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-kube-api-access-mtkrk\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059839    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-konnectivity-ca\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-socket-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-cni-netd\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjrq\" (UniqueName: \"kubernetes.io/projected/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-kube-api-access-znjrq\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-config\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-multus-daemon-config\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-k8s-cni-cncf-io\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-k8s-cni-cncf-io\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-iptables-alerter-script\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-slash\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.059995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf26828-2a89-47d0-abed-a969ada126d1-socket-dir\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-ovn\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-node-log\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-multus\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-host-slash\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.061670 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-agent-certs\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-run-ovn\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a449619b-0633-4b78-9196-cd078b370979-cni-binary-copy\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-node-log\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-multus\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-tmp-dir\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-bin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-multus-certs\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-script-lib\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-hosts-file\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-var-lib-cni-bin\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-hosts-file\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a449619b-0633-4b78-9196-cd078b370979-host-run-multus-certs\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.060429    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.060484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:05.560467036 +0000 UTC m=+3.101960192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-iptables-alerter-script\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.062517 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-tmp-dir\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.063406 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.060739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovnkube-script-lib\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.063406 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.062982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.063406 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.063148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6662a4b-5ab9-4de4-acc5-8d8cf095e13c-agent-certs\") pod \"konnectivity-agent-ch8lh\" (UID: \"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c\") " pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:05.068092 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.068064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qvq\" (UniqueName: \"kubernetes.io/projected/baf26828-2a89-47d0-abed-a969ada126d1-kube-api-access-98qvq\") pod \"aws-ebs-csi-driver-node-wbrkf\" (UID: \"baf26828-2a89-47d0-abed-a969ada126d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.068864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.068841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcs9h\" (UniqueName: \"kubernetes.io/projected/68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b-kube-api-access-fcs9h\") pod \"iptables-alerter-fl6pd\" (UID: \"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b\") " pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.069030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.068922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pbz\" (UniqueName: \"kubernetes.io/projected/a449619b-0633-4b78-9196-cd078b370979-kube-api-access-l6pbz\") pod \"multus-stpwt\" (UID: \"a449619b-0633-4b78-9196-cd078b370979\") " pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.072113 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.072081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjrq\" (UniqueName: \"kubernetes.io/projected/373d37ba-f50e-4fc6-bf05-56f1c09dce5f-kube-api-access-znjrq\") pod \"node-resolver-tsjtt\" (UID: \"373d37ba-f50e-4fc6-bf05-56f1c09dce5f\") " pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.072365 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.072346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkrk\" (UniqueName: \"kubernetes.io/projected/27a85ec8-0b6d-4019-b0aa-dae56d32a0d9-kube-api-access-mtkrk\") pod \"ovnkube-node-lfxnl\" (UID: \"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.151572 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.151494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmhfd"
Aug 04 22:21:05.159929 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.159902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4z68c"
Aug 04 22:21:05.170755 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.170730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89svx"
Aug 04 22:21:05.176581 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.176559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-stpwt"
Aug 04 22:21:05.185272 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.185252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:05.201734 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.201704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fl6pd"
Aug 04 22:21:05.210021 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.210002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf"
Aug 04 22:21:05.219000 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.218980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:05.226245 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.226215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsjtt"
Aug 04 22:21:05.463636 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.463535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:05.463804 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.463679    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:05.463804 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.463767    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:06.463745121 +0000 UTC m=+4.005238288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:05.564764 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.564728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:05.564962 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.564782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:05.564962 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.564908    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:05.564962 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.564896    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:05.564962 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.564949    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:05.564962 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.564961    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:05.565215 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.564996    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:06.564971036 +0000 UTC m=+4.106464212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:05.565215 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:05.565017    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:06.565007918 +0000 UTC m=+4.106501072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:05.889074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.888982    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 22:16:05 +0000 UTC" deadline="2028-05-13 01:27:07.412234152 +0000 UTC"
Aug 04 22:21:05.889074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:05.889020    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15531h6m1.523219091s"
Aug 04 22:21:06.023467 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:06.023398    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6662a4b_5ab9_4de4_acc5_8d8cf095e13c.slice/crio-82b0e5b89ed006cc23acb5a5c72ef9d50e5f5fcee0ae75cba18fa573c7dc034f WatchSource:0}: Error finding container 82b0e5b89ed006cc23acb5a5c72ef9d50e5f5fcee0ae75cba18fa573c7dc034f: Status 404 returned error can't find the container with id 82b0e5b89ed006cc23acb5a5c72ef9d50e5f5fcee0ae75cba18fa573c7dc034f
Aug 04 22:21:06.026350 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:06.026197    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda449619b_0633_4b78_9196_cd078b370979.slice/crio-319ef46604b6e3c0d29a23f7de6b6702f63698fa51b9c405bc69eb60d96ce18e WatchSource:0}: Error finding container 319ef46604b6e3c0d29a23f7de6b6702f63698fa51b9c405bc69eb60d96ce18e: Status 404 returned error can't find the container with id 319ef46604b6e3c0d29a23f7de6b6702f63698fa51b9c405bc69eb60d96ce18e
Aug 04 22:21:06.029703 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:06.029412    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b4b1ee_d632_43ea_9ac8_83eb4b98ef0b.slice/crio-e31df95d0a56e82efdfe7bd1711e8a0ab2cbb6f655ada724007bf7507b728aa0 WatchSource:0}: Error finding container e31df95d0a56e82efdfe7bd1711e8a0ab2cbb6f655ada724007bf7507b728aa0: Status 404 returned error can't find the container with id e31df95d0a56e82efdfe7bd1711e8a0ab2cbb6f655ada724007bf7507b728aa0
Aug 04 22:21:06.030198 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:06.030163    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a85ec8_0b6d_4019_b0aa_dae56d32a0d9.slice/crio-308fbb789e09774ea04c1b20b71dc7718d94484b98bd0d61caf17a8581b7ad65 WatchSource:0}: Error finding container 308fbb789e09774ea04c1b20b71dc7718d94484b98bd0d61caf17a8581b7ad65: Status 404 returned error can't find the container with id 308fbb789e09774ea04c1b20b71dc7718d94484b98bd0d61caf17a8581b7ad65
Aug 04 22:21:06.038891 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:06.038866    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf26828_2a89_47d0_abed_a969ada126d1.slice/crio-51d3bde822869d2d4785e639fbe6660758096fa06c31a4de588db8447954d4b8 WatchSource:0}: Error finding container 51d3bde822869d2d4785e639fbe6660758096fa06c31a4de588db8447954d4b8: Status 404 returned error can't find the container with id 51d3bde822869d2d4785e639fbe6660758096fa06c31a4de588db8447954d4b8
Aug 04 22:21:06.473303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:06.473015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:06.473303 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.473289    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:06.473554 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.473362    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:08.473340287 +0000 UTC m=+6.014833445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:06.574482 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:06.574402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:06.574642 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:06.574466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:06.574642 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574578    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:06.574642 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574629    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:06.574802 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574652    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:06.574802 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574658    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:08.574636695 +0000 UTC m=+6.116129851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:06.574802 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574666    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:06.574802 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:06.574703    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:08.574691975 +0000 UTC m=+6.116185130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:06.889756 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:06.889627    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 22:16:05 +0000 UTC" deadline="2028-03-20 01:01:18.495952802 +0000 UTC"
Aug 04 22:21:06.889756 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:06.889695    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14234h40m11.606262333s"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.006928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:07.007095    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.007536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:07.007646    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.007712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:07.008178 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:07.007789    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:07.022783 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.022692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fl6pd" event={"ID":"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b","Type":"ContainerStarted","Data":"e31df95d0a56e82efdfe7bd1711e8a0ab2cbb6f655ada724007bf7507b728aa0"}
Aug 04 22:21:07.035312 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.035251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-stpwt" event={"ID":"a449619b-0633-4b78-9196-cd078b370979","Type":"ContainerStarted","Data":"319ef46604b6e3c0d29a23f7de6b6702f63698fa51b9c405bc69eb60d96ce18e"}
Aug 04 22:21:07.040195 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.039315    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e0ac1cd241ea2e991901d610263cbe9" containerID="57c9790a78e7f53e5ea5f8529d13ecb2582791bbc92e6bf7c9c73f08260c4dc6" exitCode=0
Aug 04 22:21:07.040195 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.039395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal" event={"ID":"2e0ac1cd241ea2e991901d610263cbe9","Type":"ContainerDied","Data":"57c9790a78e7f53e5ea5f8529d13ecb2582791bbc92e6bf7c9c73f08260c4dc6"}
Aug 04 22:21:07.061332 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.061306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmhfd" event={"ID":"604304b6-6b2a-4511-af73-402e56662794","Type":"ContainerStarted","Data":"42a978e1b352d251edfc7d6abb489be0ca8abbc21b668469253ad01565c7ad0c"}
Aug 04 22:21:07.074932 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.074886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z68c" event={"ID":"0321c2f7-349d-41ee-844b-7710cdab0e8c","Type":"ContainerStarted","Data":"647a197a6512f3531e7a82c5221fdf4d9f6bef8d9138aca9c304eb04949fe55f"}
Aug 04 22:21:07.083799 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.083767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ch8lh" event={"ID":"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c","Type":"ContainerStarted","Data":"82b0e5b89ed006cc23acb5a5c72ef9d50e5f5fcee0ae75cba18fa573c7dc034f"}
Aug 04 22:21:07.090887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.090858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal" event={"ID":"b56d4e9c7e28a671fb29fb60ed5ddf2b","Type":"ContainerStarted","Data":"7f9016769a00cedabca0e678e85961ff1c08833eb73e985de2f8f661b78de762"}
Aug 04 22:21:07.109628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.109557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf" event={"ID":"baf26828-2a89-47d0-abed-a969ada126d1","Type":"ContainerStarted","Data":"51d3bde822869d2d4785e639fbe6660758096fa06c31a4de588db8447954d4b8"}
Aug 04 22:21:07.116123 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.116050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerStarted","Data":"134e04d93461469595259816d869a37ea96cdb248cdbba153ab2aafd68208948"}
Aug 04 22:21:07.131927 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.131898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsjtt" event={"ID":"373d37ba-f50e-4fc6-bf05-56f1c09dce5f","Type":"ContainerStarted","Data":"0d2ac44ebb4e3a227f8c0acbdc652ef273d21a96f292ddcee3e4113fe6a53c70"}
Aug 04 22:21:07.147327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:07.147259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"308fbb789e09774ea04c1b20b71dc7718d94484b98bd0d61caf17a8581b7ad65"}
Aug 04 22:21:08.180607 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:08.180571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal" event={"ID":"2e0ac1cd241ea2e991901d610263cbe9","Type":"ContainerStarted","Data":"afc3556dba4d1ef3967c0654e4ecab581a88e833b5a25b6cab808204e78fbf5f"}
Aug 04 22:21:08.195219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:08.195162    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-146.ec2.internal" podStartSLOduration=4.195140919 podStartE2EDuration="4.195140919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:07.105208819 +0000 UTC m=+4.646701997" watchObservedRunningTime="2026-08-04 22:21:08.195140919 +0000 UTC m=+5.736634099"
Aug 04 22:21:08.490289 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:08.490198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:08.490505 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.490412    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:08.490505 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.490479    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:12.49045805 +0000 UTC m=+10.031951220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:08.591466 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:08.591424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:08.591466 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:08.591491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:08.591739 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.591689    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:08.591739 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.591713    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:08.591739 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.591727    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:08.592000 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.591787    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:12.59176693 +0000 UTC m=+10.133260087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:08.592304 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.592275    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:08.592378 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:08.592345    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:12.592327141 +0000 UTC m=+10.133820300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:09.010779 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:09.010747    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:09.010954 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:09.010890    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:09.011360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:09.011272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:09.011457 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:09.011370    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:09.011457 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:09.011421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:09.011572 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:09.011505    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:11.005190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:11.005106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:11.005190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:11.005147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:11.005677 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:11.005106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:11.005677 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:11.005275    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:11.005677 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:11.005337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:11.005677 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:11.005411    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:12.525296 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:12.525260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:12.525721 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.525430    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:12.525721 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.525520    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:20.525497671 +0000 UTC m=+18.066990831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:12.626429 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:12.626390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:12.626575 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:12.626451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:12.626634 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.626587    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:12.626634 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.626603    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:12.626634 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.626613    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:12.626734 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.626658    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:20.626645836 +0000 UTC m=+18.168138991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:12.627022 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.627001    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:12.627172 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:12.627043    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:20.627032601 +0000 UTC m=+18.168525755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:13.009617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:13.009532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:13.009788 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:13.009648    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:13.010062 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:13.010035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:13.010184 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:13.010142    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:13.010313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:13.010198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:13.010313 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:13.010289    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:15.008384 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:15.008310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:15.008384 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:15.008353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:15.008762 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:15.008430    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:15.008762 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:15.008482    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:15.008762 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:15.008544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:15.008762 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:15.008625    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:17.007835 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:17.007802    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:17.007835 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:17.007825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:17.008295 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:17.007815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:17.008295 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:17.007946    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:17.008295 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:17.008062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:17.008295 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:17.008185    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:19.004474 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:19.004437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:19.004474 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:19.004456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:19.004940 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:19.004566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:19.004940 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:19.004558    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:19.004940 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:19.004659    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:19.004940 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:19.004733    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:20.589074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:20.589030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:20.589607 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.589189    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:20.589607 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.589268    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:36.589248094 +0000 UTC m=+34.130741262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:20.689883 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:20.689846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:20.690083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:20.689956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:20.690083 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690029    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:20.690083 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690053    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:20.690083 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690060    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:20.690083 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690065    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:20.690307 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690126    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:36.690106836 +0000 UTC m=+34.231599989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:20.690307 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:20.690144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:36.690135251 +0000 UTC m=+34.231628406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:21.007330 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:21.007302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:21.007330 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:21.007329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:21.007532 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:21.007413    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:21.007532 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:21.007462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:21.007653 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:21.007560    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:21.007704 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:21.007671    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:23.004947 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:23.004910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:23.005415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:23.005001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:23.005415 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:23.005028    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:23.005415 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:23.005140    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:23.005415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:23.005168    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:23.005415 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:23.005266    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:25.009267 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.009241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:25.009684 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:25.009359    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:25.009684 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.009677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:25.009792 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:25.009768    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:25.009846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.009820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:25.009939 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:25.009919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:25.220680 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.220158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4z68c" event={"ID":"0321c2f7-349d-41ee-844b-7710cdab0e8c","Type":"ContainerStarted","Data":"1da40b1255d5e9c9547e316573f70fa8926142f47d2db59d47540081435d1db1"}
Aug 04 22:21:25.222060 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.222005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ch8lh" event={"ID":"c6662a4b-5ab9-4de4-acc5-8d8cf095e13c","Type":"ContainerStarted","Data":"0ab295fd5562c6f16ffc2c05c6955e38ca2f3e2cb3d974ab3b2df1baeebb2df8"}
Aug 04 22:21:25.224256 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.224225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf" event={"ID":"baf26828-2a89-47d0-abed-a969ada126d1","Type":"ContainerStarted","Data":"d6b7d18dde08dc4faf23b5e27374ad237372ac086de3774a9194634886cabf53"}
Aug 04 22:21:25.225794 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.225753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerStarted","Data":"70a41dd0d5fe0b5fb0eb84478e148abde54aca18f08c6ab77a70ac3b206c9456"}
Aug 04 22:21:25.227451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.227405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsjtt" event={"ID":"373d37ba-f50e-4fc6-bf05-56f1c09dce5f","Type":"ContainerStarted","Data":"fcb934d57460b11bdcdb1bc011b4800d4df65115d07116956e8f7d90547753b7"}
Aug 04 22:21:25.228860 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.228832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"75d7034def0e09c12832a1bf176824128593dbaaff42920c19f9295ff7ae91f3"}
Aug 04 22:21:25.231141 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.230986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-stpwt" event={"ID":"a449619b-0633-4b78-9196-cd078b370979","Type":"ContainerStarted","Data":"d907d2f6a5e68f5aba30e42deb26ca79096032335066dd809edbe28671665269"}
Aug 04 22:21:25.232714 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.232674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmhfd" event={"ID":"604304b6-6b2a-4511-af73-402e56662794","Type":"ContainerStarted","Data":"6048e08b5ff411586b7afb59ba708a3a82f87c50dbb2117f28a9df455991f51f"}
Aug 04 22:21:25.234604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.234111    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-146.ec2.internal" podStartSLOduration=21.234093822 podStartE2EDuration="21.234093822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:08.196064572 +0000 UTC m=+5.737557752" watchObservedRunningTime="2026-08-04 22:21:25.234093822 +0000 UTC m=+22.775586997"
Aug 04 22:21:25.234706 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.234654    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4z68c" podStartSLOduration=3.296004711 podStartE2EDuration="22.234641534s" totalImagesPullingTime="18.938636823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:25.233521443 +0000 UTC m=+22.775014637" watchObservedRunningTime="2026-08-04 22:21:25.234641534 +0000 UTC m=+22.776134711"
Aug 04 22:21:25.263541 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.263253    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ch8lh" podStartSLOduration=3.323508466 podStartE2EDuration="22.263236554s" totalImagesPullingTime="18.939728088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:25.246158027 +0000 UTC m=+22.787651210" watchObservedRunningTime="2026-08-04 22:21:25.263236554 +0000 UTC m=+22.804729731"
Aug 04 22:21:25.279521 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.279474    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lmhfd" podStartSLOduration=3.337744356 podStartE2EDuration="22.279459102s" totalImagesPullingTime="18.941714746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:25.278887009 +0000 UTC m=+22.820380194" watchObservedRunningTime="2026-08-04 22:21:25.279459102 +0000 UTC m=+22.820952275"
Aug 04 22:21:25.279672 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.279614    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tsjtt" podStartSLOduration=3.349372725 podStartE2EDuration="22.279599878s" totalImagesPullingTime="18.930227153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:25.26281859 +0000 UTC m=+22.804311764" watchObservedRunningTime="2026-08-04 22:21:25.279599878 +0000 UTC m=+22.821093053"
Aug 04 22:21:25.312433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:25.312140    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-stpwt" podStartSLOduration=3.311487498 podStartE2EDuration="22.312120847s" totalImagesPullingTime="19.000633349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:25.31201662 +0000 UTC m=+22.853509817" watchObservedRunningTime="2026-08-04 22:21:25.312120847 +0000 UTC m=+22.853614011"
Aug 04 22:21:26.235994 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.235963    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="70a41dd0d5fe0b5fb0eb84478e148abde54aca18f08c6ab77a70ac3b206c9456" exitCode=0
Aug 04 22:21:26.236706 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.236046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"70a41dd0d5fe0b5fb0eb84478e148abde54aca18f08c6ab77a70ac3b206c9456"}
Aug 04 22:21:26.241910 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.241842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"99dcb2338e8d1624fdf849ab61c869f7a71542b726de3630d534fc0e08b4a1e5"}
Aug 04 22:21:26.242018 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.241954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"9fd416bfad1ecbdc7d39c2b3cc0d4d0002fea7680f1736d797d5689c41d0bcc6"}
Aug 04 22:21:26.242018 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.241994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"d7640cf3d8569894abee1b9ba28ff9b451b5227a09ed00ceb60736882195ab39"}
Aug 04 22:21:26.242109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.242025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"341859ec60d089834653d2b26dab46ca5baa82f59de4e30908a9a90b0eb8f1a8"}
Aug 04 22:21:26.242109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.242060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"a826a4f8cc12f9af2ab01f592fdacab282e964a3e26d80cf7fed3534d48a4454"}
Aug 04 22:21:26.243165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.243136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fl6pd" event={"ID":"68b4b1ee-d632-43ea-9ac8-83eb4b98ef0b","Type":"ContainerStarted","Data":"f4e702505c50f614025141d071b66fb796a2c4025fbf17c6b13aa243a6a15087"}
Aug 04 22:21:26.272533 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.272494    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fl6pd" podStartSLOduration=4.328027183 podStartE2EDuration="23.272482698s" totalImagesPullingTime="18.944455515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:26.272231108 +0000 UTC m=+23.813724283" watchObservedRunningTime="2026-08-04 22:21:26.272482698 +0000 UTC m=+23.813975872"
Aug 04 22:21:26.428392 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.428328    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 22:21:26.919304 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.919186    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T22:21:26.428343686Z","UUID":"d1ae11e1-6f54-48fe-af26-caf3de5733fa","Handler":null,"Name":"","Endpoint":""}
Aug 04 22:21:26.920944 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.920871    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 22:21:26.920944 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:26.920941    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 22:21:27.004762 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:27.004740    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:27.004943 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:27.004851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:27.004943 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:27.004861    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:27.005052 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:27.004965    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:27.005052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:27.004977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:27.005114 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:27.005071    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:27.247975 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:27.247881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf" event={"ID":"baf26828-2a89-47d0-abed-a969ada126d1","Type":"ContainerStarted","Data":"8835f02ed4aadccfe78e1fea51cbc1e5626186f09143fb2f4826ecefb0da31f4"}
Aug 04 22:21:28.253194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:28.253151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf" event={"ID":"baf26828-2a89-47d0-abed-a969ada126d1","Type":"ContainerStarted","Data":"42bfbe7cd4fd02bbae8239575a63cec965b194ca5993fe1a9cd6c216532ed215"}
Aug 04 22:21:28.256682 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:28.256650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"01cb10f604611bfbd712749471233a8e25d964345fed257e088ebc973164a77a"}
Aug 04 22:21:29.005002 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.004966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:29.005002 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.005003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:29.005238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.004967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:29.005238 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:29.005100    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:29.005238 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:29.005145    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:29.005238 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:29.005211    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:29.217313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.217277    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:29.218113 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.218091    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:29.231390 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.231345    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbrkf" podStartSLOduration=5.1275881309999995 podStartE2EDuration="26.231329926s" totalImagesPullingTime="21.103741795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:28.272947727 +0000 UTC m=+25.814440901" watchObservedRunningTime="2026-08-04 22:21:29.231329926 +0000 UTC m=+26.772823094"
Aug 04 22:21:29.259822 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.259752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:29.260217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.259977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ch8lh"
Aug 04 22:21:29.344553 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.344526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:29.421549 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:29.421515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:31.004335 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.004164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:31.005017 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.004170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:31.005017 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:31.004402    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:31.005017 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.004234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:31.005017 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:31.004483    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:31.005017 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:31.004583    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:31.265566 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.265493    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="2888d8a9e34346bfaeeb914540c3cc6923e9b5508fc429af94d01102e3c97c15" exitCode=0
Aug 04 22:21:31.265762 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.265582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"2888d8a9e34346bfaeeb914540c3cc6923e9b5508fc429af94d01102e3c97c15"}
Aug 04 22:21:31.269512 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.269485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" event={"ID":"27a85ec8-0b6d-4019-b0aa-dae56d32a0d9","Type":"ContainerStarted","Data":"d3dad3b8df48b988943a2e7ae399aafd5e8a2f40f5805366b4fa0808d97e8471"}
Aug 04 22:21:31.309812 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:31.309761    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl" podStartSLOduration=9.303494515 podStartE2EDuration="28.30974865s" totalImagesPullingTime="19.006254135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:31.309445576 +0000 UTC m=+28.850938750" watchObservedRunningTime="2026-08-04 22:21:31.30974865 +0000 UTC m=+28.851241824"
Aug 04 22:21:32.273823 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.273632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerStarted","Data":"b36b7d3e6d6275fc37f5c205edaeeb11f4c401042cbaeea7566e240fb0f652ec"}
Aug 04 22:21:32.274193 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.274159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:32.274254 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.274228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:32.274301 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.274265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:32.289356 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.289338    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:32.289887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.289870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:21:32.666528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.666502    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5855j"]
Aug 04 22:21:32.666735 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.666627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:32.666735 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:32.666726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:32.667266 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.667247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wxcj"]
Aug 04 22:21:32.667349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.667339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:32.667453 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:32.667429    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:32.669067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.669043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqcgb"]
Aug 04 22:21:32.669162 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:32.669149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:32.669269 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:32.669248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:33.277841 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:33.277807    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="b36b7d3e6d6275fc37f5c205edaeeb11f4c401042cbaeea7566e240fb0f652ec" exitCode=0
Aug 04 22:21:33.278320 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:33.277885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"b36b7d3e6d6275fc37f5c205edaeeb11f4c401042cbaeea7566e240fb0f652ec"}
Aug 04 22:21:33.326445 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:33.326424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:34.004445 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:34.004419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:34.004565 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:34.004422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:34.004565 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:34.004523    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:34.004642 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:34.004595    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:35.004255 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:35.004221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:35.004645 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:35.004331    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:35.284665 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:35.284452    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="673dbf5fafe9ae0898366f3f7424972a6b02eb03b75164d1a3b75f01d208a955" exitCode=0
Aug 04 22:21:35.284665 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:35.284528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"673dbf5fafe9ae0898366f3f7424972a6b02eb03b75164d1a3b75f01d208a955"}
Aug 04 22:21:35.918317 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:35.918282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:35.972253 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:35.972224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:36.005083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.005060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:36.005488 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.005062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:36.005488 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.005169    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:36.005488 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.005268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:36.607268 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.607178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:36.607437 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.607316    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:36.607498 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.607472    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:22:08.607449485 +0000 UTC m=+66.148942642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 22:21:36.707947 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.707914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:36.708103 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.707966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:36.708103 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708086    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:36.708237 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708123    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 22:21:36.708237 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708139    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 22:21:36.708237 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708152    2582 projected.go:196] Error preparing data for projected volume kube-api-access-cgtdf for pod openshift-network-diagnostics/network-check-target-2wxcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:36.708237 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708161    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret podName:05b4f6e1-6a4e-4e9b-9415-5e86cbca6167 nodeName:}" failed. No retries permitted until 2026-08-04 22:22:08.708142132 +0000 UTC m=+66.249635298 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret") pod "global-pull-secret-syncer-5855j" (UID: "05b4f6e1-6a4e-4e9b-9415-5e86cbca6167") : object "kube-system"/"original-pull-secret" not registered
Aug 04 22:21:36.708237 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:36.708193    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf podName:cdb3a868-bc9e-4925-ba14-a1a076c4dcd8 nodeName:}" failed. No retries permitted until 2026-08-04 22:22:08.708181997 +0000 UTC m=+66.249675148 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cgtdf" (UniqueName: "kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf") pod "network-check-target-2wxcj" (UID: "cdb3a868-bc9e-4925-ba14-a1a076c4dcd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 22:21:36.919097 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.919049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:36.985576 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:36.985538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:37.004922 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:37.004628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:37.004922 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:37.004771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wxcj" podUID="cdb3a868-bc9e-4925-ba14-a1a076c4dcd8"
Aug 04 22:21:37.123617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:37.123589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:37.183190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:37.183120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:37.856938 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:37.856890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:37.914045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:37.914014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:38.004654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.004623    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:38.004802 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:38.004750    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5855j" podUID="05b4f6e1-6a4e-4e9b-9415-5e86cbca6167"
Aug 04 22:21:38.004854 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.004803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:38.004955 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:38.004933    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:21:38.759027 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.758955    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-146.ec2.internal" event="NodeReady"
Aug 04 22:21:38.759440 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.759069    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 22:21:38.805813 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.805784    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fggh6"]
Aug 04 22:21:38.808333 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.808306    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qqhwj"]
Aug 04 22:21:38.808470 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.808451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:38.810168 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.810145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:38.811258 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.811233    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 22:21:38.811408 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.811282    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d46fp\""
Aug 04 22:21:38.811408 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.811242    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 22:21:38.812434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.812416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 22:21:38.812934 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.812886    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 22:21:38.813054 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.813025    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qld8r\""
Aug 04 22:21:38.813453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.813432    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 22:21:38.829077 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.829055    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fggh6"]
Aug 04 22:21:38.832164 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.832137    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqhwj"]
Aug 04 22:21:38.925961 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.925926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:38.926145 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.925974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-cert\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:38.926145 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.926002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ddc17-b853-463e-9a50-720ea2758229-config-volume\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:38.926145 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.926045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822h5\" (UniqueName: \"kubernetes.io/projected/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-kube-api-access-822h5\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:38.926145 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.926103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68ddc17-b853-463e-9a50-720ea2758229-tmp-dir\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:38.926355 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:38.926149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68fk\" (UniqueName: \"kubernetes.io/projected/e68ddc17-b853-463e-9a50-720ea2758229-kube-api-access-j68fk\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.004705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.004669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:21:39.007183 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.007118    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 22:21:39.007183 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.007160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 22:21:39.007183 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.007174    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j2l9z\""
Aug 04 22:21:39.027088 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.027205 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-cert\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:39.027205 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ddc17-b853-463e-9a50-720ea2758229-config-volume\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.027205 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822h5\" (UniqueName: \"kubernetes.io/projected/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-kube-api-access-822h5\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:39.027205 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68ddc17-b853-463e-9a50-720ea2758229-tmp-dir\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.027424 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:39.027212    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:39.027424 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.027229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68fk\" (UniqueName: \"kubernetes.io/projected/e68ddc17-b853-463e-9a50-720ea2758229-kube-api-access-j68fk\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.027424 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:39.027272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:39.527252431 +0000 UTC m=+37.068745589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:39.028210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.028192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ddc17-b853-463e-9a50-720ea2758229-config-volume\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.028304 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.028264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68ddc17-b853-463e-9a50-720ea2758229-tmp-dir\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.032146 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.032129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-cert\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:39.035018 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.034998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68fk\" (UniqueName: \"kubernetes.io/projected/e68ddc17-b853-463e-9a50-720ea2758229-kube-api-access-j68fk\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.035410 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.035369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822h5\" (UniqueName: \"kubernetes.io/projected/8fbf8a0c-ad84-445a-beab-7a2f37afd2a7-kube-api-access-822h5\") pod \"ingress-canary-qqhwj\" (UID: \"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7\") " pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:39.126402 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.126370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqhwj"
Aug 04 22:21:39.529849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.529775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:39.529982 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:39.529963    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:39.530035 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:39.530025    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:40.530008726 +0000 UTC m=+38.071501878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:39.558216 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:39.558193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqhwj"]
Aug 04 22:21:39.560817 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:21:39.560792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbf8a0c_ad84_445a_beab_7a2f37afd2a7.slice/crio-6115f7ee109bcf3ba656ebb0f3050310501cce2dd6836948cdac6fe14d451cf1 WatchSource:0}: Error finding container 6115f7ee109bcf3ba656ebb0f3050310501cce2dd6836948cdac6fe14d451cf1: Status 404 returned error can't find the container with id 6115f7ee109bcf3ba656ebb0f3050310501cce2dd6836948cdac6fe14d451cf1
Aug 04 22:21:40.004604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.004577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:21:40.005414 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.004619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:21:40.007108 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.007082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 22:21:40.008026 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.008003    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 22:21:40.008132 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.008004    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5596r\""
Aug 04 22:21:40.299208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.299125    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="1eb4c6f748a08b894da54c454faaf0fb577a86433789473218f501820abe20f3" exitCode=0
Aug 04 22:21:40.299336 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.299208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"1eb4c6f748a08b894da54c454faaf0fb577a86433789473218f501820abe20f3"}
Aug 04 22:21:40.300433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.300406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqhwj" event={"ID":"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7","Type":"ContainerStarted","Data":"6115f7ee109bcf3ba656ebb0f3050310501cce2dd6836948cdac6fe14d451cf1"}
Aug 04 22:21:40.538414 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:40.538375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:40.538568 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:40.538538    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:40.538635 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:40.538623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:42.538603195 +0000 UTC m=+40.080096355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:41.306242 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:41.306208    2582 generic.go:356] "Generic (PLEG): container finished" podID="a38d7069-0118-40fd-ba16-aaffbb74a00c" containerID="61052dc54120ed64e0d7e490f6fb2d6f33661cbd3ec92f58e1a1048ffb6185ba" exitCode=0
Aug 04 22:21:41.306704 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:41.306275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerDied","Data":"61052dc54120ed64e0d7e490f6fb2d6f33661cbd3ec92f58e1a1048ffb6185ba"}
Aug 04 22:21:42.311935 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.311641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89svx" event={"ID":"a38d7069-0118-40fd-ba16-aaffbb74a00c","Type":"ContainerStarted","Data":"5ef6ff9330e2d7d15a5fa1d4da34441c800b88da9d2f6de3b196d496b95cee24"}
Aug 04 22:21:42.313110 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.313085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqhwj" event={"ID":"8fbf8a0c-ad84-445a-beab-7a2f37afd2a7","Type":"ContainerStarted","Data":"3e6b6d219f329fcac5a6966c3c01adeadfe322da5a52da7ea0ca257bf7912b75"}
Aug 04 22:21:42.332391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.332350    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-89svx" podStartSLOduration=10.16906197 podStartE2EDuration="39.33233835s" totalImagesPullingTime="29.16327638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:42.331157946 +0000 UTC m=+39.872651120" watchObservedRunningTime="2026-08-04 22:21:42.33233835 +0000 UTC m=+39.873831524"
Aug 04 22:21:42.344476 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.344437    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qqhwj" podStartSLOduration=2.012954954 podStartE2EDuration="4.34442501s" totalImagesPullingTime="2.331470056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:21:42.344179881 +0000 UTC m=+39.885673055" watchObservedRunningTime="2026-08-04 22:21:42.34442501 +0000 UTC m=+39.885918184"
Aug 04 22:21:42.421274 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.421244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:42.521560 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.521537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:42.555024 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:42.555000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:42.555096 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:42.555074    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:42.555139 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:42.555117    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:46.555106161 +0000 UTC m=+44.096599313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:43.279806 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:43.279778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:43.330170 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:43.330140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:46.582053 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:46.582020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:46.582395 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:46.582138    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:46.582395 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:46.582190    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:21:54.582176361 +0000 UTC m=+52.123669512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:53.889431 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:53.889399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:53.991710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:53.991677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:54.635106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:54.635067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:21:54.635266 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:54.635190    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:21:54.635266 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:21:54.635248    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:22:10.635230345 +0000 UTC m=+68.176723690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:21:59.085187 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:59.085151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:21:59.141910 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:21:59.141875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:04.292150 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:04.292114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lfxnl"
Aug 04 22:22:04.428552 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:04.428518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:04.528911 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:04.528842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:04.578514 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:04.578454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:08.043067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.043030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:08.141058 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.141024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:08.625611 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.625571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:22:08.628013 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.627997    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 22:22:08.635948 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:08.635931    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 22:22:08.635990 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:08.635983    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:23:12.635967748 +0000 UTC m=+130.177460900 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : secret "metrics-daemon-secret" not found
Aug 04 22:22:08.726656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.726626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:22:08.726721 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.726667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:22:08.729304 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.729271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 22:22:08.729345 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.729271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 22:22:08.738969 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.738945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 22:22:08.741022 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.741003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05b4f6e1-6a4e-4e9b-9415-5e86cbca6167-original-pull-secret\") pod \"global-pull-secret-syncer-5855j\" (UID: \"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167\") " pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:22:08.750030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.750007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtdf\" (UniqueName: \"kubernetes.io/projected/cdb3a868-bc9e-4925-ba14-a1a076c4dcd8-kube-api-access-cgtdf\") pod \"network-check-target-2wxcj\" (UID: \"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8\") " pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:22:08.816075 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.816054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5855j"
Aug 04 22:22:08.938560 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:08.938530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5855j"]
Aug 04 22:22:08.941512 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:22:08.941486    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b4f6e1_6a4e_4e9b_9415_5e86cbca6167.slice/crio-556dd15e116043389fb281b25427064df8d9b590fc7ce9d8e7fecd558bfa7822 WatchSource:0}: Error finding container 556dd15e116043389fb281b25427064df8d9b590fc7ce9d8e7fecd558bfa7822: Status 404 returned error can't find the container with id 556dd15e116043389fb281b25427064df8d9b590fc7ce9d8e7fecd558bfa7822
Aug 04 22:22:09.020022 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:09.019997    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j2l9z\""
Aug 04 22:22:09.028690 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:09.028670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:22:09.159240 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:09.159162    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wxcj"]
Aug 04 22:22:09.162635 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:22:09.162613    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb3a868_bc9e_4925_ba14_a1a076c4dcd8.slice/crio-e73840dd4da3409cb3f819860ddf4f5c0f918d0301be9a2b46d2878f683224bb WatchSource:0}: Error finding container e73840dd4da3409cb3f819860ddf4f5c0f918d0301be9a2b46d2878f683224bb: Status 404 returned error can't find the container with id e73840dd4da3409cb3f819860ddf4f5c0f918d0301be9a2b46d2878f683224bb
Aug 04 22:22:09.373759 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:09.373728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wxcj" event={"ID":"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8","Type":"ContainerStarted","Data":"e73840dd4da3409cb3f819860ddf4f5c0f918d0301be9a2b46d2878f683224bb"}
Aug 04 22:22:09.374805 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:09.374780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5855j" event={"ID":"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167","Type":"ContainerStarted","Data":"556dd15e116043389fb281b25427064df8d9b590fc7ce9d8e7fecd558bfa7822"}
Aug 04 22:22:10.640696 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:10.640652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:22:10.641158 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:10.640819    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:22:10.641158 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:10.640949    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:22:42.640926337 +0000 UTC m=+100.182419493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:22:14.389047 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:14.389003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wxcj" event={"ID":"cdb3a868-bc9e-4925-ba14-a1a076c4dcd8","Type":"ContainerStarted","Data":"3438d580941f383169ec78ddd70e4f1351e9e5b6029109c198ce039a9704c855"}
Aug 04 22:22:14.389505 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:14.389097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:22:14.390492 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:14.390469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5855j" event={"ID":"05b4f6e1-6a4e-4e9b-9415-5e86cbca6167","Type":"ContainerStarted","Data":"1fef46a7c5459a9065d2b0282b2fac4bcf7ea90afeafbd85ec0d3015a3562d5e"}
Aug 04 22:22:14.402782 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:14.402736    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2wxcj" podStartSLOduration=66.914331345 podStartE2EDuration="1m11.402724018s" totalImagesPullingTime="4.488392673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:22:14.402605973 +0000 UTC m=+71.944099148" watchObservedRunningTime="2026-08-04 22:22:14.402724018 +0000 UTC m=+71.944217183"
Aug 04 22:22:14.415872 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:14.415828    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5855j" podStartSLOduration=66.7029553 podStartE2EDuration="1m11.415816099s" totalImagesPullingTime="4.712860799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:22:14.415334926 +0000 UTC m=+71.956828100" watchObservedRunningTime="2026-08-04 22:22:14.415816099 +0000 UTC m=+71.957309274"
Aug 04 22:22:23.008855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.008748    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"]
Aug 04 22:22:23.010737 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.010722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.013270 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.013243    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 22:22:23.014048 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.014030    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 04 22:22:23.014161 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.014050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 22:22:23.014161 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.014030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 22:22:23.025672 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.025648    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"]
Aug 04 22:22:23.126083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.126050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e843ed2b-5d51-439b-ad6e-317eca36a388-tmp\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.126083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.126085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glt9b\" (UniqueName: \"kubernetes.io/projected/e843ed2b-5d51-439b-ad6e-317eca36a388-kube-api-access-glt9b\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.126282 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.126115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e843ed2b-5d51-439b-ad6e-317eca36a388-klusterlet-config\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.226661 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.226627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e843ed2b-5d51-439b-ad6e-317eca36a388-tmp\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.226661 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.226666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glt9b\" (UniqueName: \"kubernetes.io/projected/e843ed2b-5d51-439b-ad6e-317eca36a388-kube-api-access-glt9b\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.226832 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.226695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e843ed2b-5d51-439b-ad6e-317eca36a388-klusterlet-config\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.227104 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.227087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e843ed2b-5d51-439b-ad6e-317eca36a388-tmp\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.229213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.229192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e843ed2b-5d51-439b-ad6e-317eca36a388-klusterlet-config\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.234846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.234827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glt9b\" (UniqueName: \"kubernetes.io/projected/e843ed2b-5d51-439b-ad6e-317eca36a388-kube-api-access-glt9b\") pod \"klusterlet-addon-workmgr-5656bdbdc8-4nlxn\" (UID: \"e843ed2b-5d51-439b-ad6e-317eca36a388\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.320083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.320000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:23.439604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:23.439536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"]
Aug 04 22:22:23.443477 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:22:23.443438    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode843ed2b_5d51_439b_ad6e_317eca36a388.slice/crio-05c7cee726a1634abe8b9e3d6ae0d52acc0e7c82a1d157ed2e4a152763b49f8c WatchSource:0}: Error finding container 05c7cee726a1634abe8b9e3d6ae0d52acc0e7c82a1d157ed2e4a152763b49f8c: Status 404 returned error can't find the container with id 05c7cee726a1634abe8b9e3d6ae0d52acc0e7c82a1d157ed2e4a152763b49f8c
Aug 04 22:22:24.417555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:24.417514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn" event={"ID":"e843ed2b-5d51-439b-ad6e-317eca36a388","Type":"ContainerStarted","Data":"05c7cee726a1634abe8b9e3d6ae0d52acc0e7c82a1d157ed2e4a152763b49f8c"}
Aug 04 22:22:29.432772 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:29.432730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn" event={"ID":"e843ed2b-5d51-439b-ad6e-317eca36a388","Type":"ContainerStarted","Data":"0141ede8d55986cee2e5e4be1f58b258ec42245c9b471f739d3bffdd24626071"}
Aug 04 22:22:29.433174 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:29.433024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:29.434360 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:29.434339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:22:29.447420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:29.447365    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn" podStartSLOduration=2.51180702 podStartE2EDuration="7.447349062s" totalImagesPullingTime="4.935542042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:22:29.447117045 +0000 UTC m=+86.988610222" watchObservedRunningTime="2026-08-04 22:22:29.447349062 +0000 UTC m=+86.988842238"
Aug 04 22:22:31.083204 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:31.083153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:31.184354 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:31.184323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:22:42.667105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:42.667068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:22:42.667570 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:42.667192    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:22:42.667570 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:22:42.667272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:23:46.66725172 +0000 UTC m=+164.208744892 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:22:45.395347 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:22:45.395312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2wxcj"
Aug 04 22:23:12.664985 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:23:12.664937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:23:12.665467 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:12.665071    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 22:23:12.665467 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:12.665150    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:14.665133054 +0000 UTC m=+252.206626206 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : secret "metrics-daemon-secret" not found
Aug 04 22:23:41.819983 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:41.819934    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fggh6" podUID="e68ddc17-b853-463e-9a50-720ea2758229"
Aug 04 22:23:42.602501 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:23:42.602469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:23:43.022069 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:43.022034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:23:46.684835 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:23:46.684801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:23:46.685228 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:46.684934    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 22:23:46.685228 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:23:46.684998    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls podName:e68ddc17-b853-463e-9a50-720ea2758229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:48.684984705 +0000 UTC m=+286.226477858 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls") pod "dns-default-fggh6" (UID: "e68ddc17-b853-463e-9a50-720ea2758229") : secret "dns-default-metrics-tls" not found
Aug 04 22:23:54.004759 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:23:54.004687    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:24:28.710198 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:28.710161    2582 generic.go:356] "Generic (PLEG): container finished" podID="e843ed2b-5d51-439b-ad6e-317eca36a388" containerID="0141ede8d55986cee2e5e4be1f58b258ec42245c9b471f739d3bffdd24626071" exitCode=1
Aug 04 22:24:28.710590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:28.710211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn" event={"ID":"e843ed2b-5d51-439b-ad6e-317eca36a388","Type":"ContainerDied","Data":"0141ede8d55986cee2e5e4be1f58b258ec42245c9b471f739d3bffdd24626071"}
Aug 04 22:24:28.710590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:28.710490    2582 scope.go:117] "RemoveContainer" containerID="0141ede8d55986cee2e5e4be1f58b258ec42245c9b471f739d3bffdd24626071"
Aug 04 22:24:29.433290 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:29.433237    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:24:29.714212 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:29.714121    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn" event={"ID":"e843ed2b-5d51-439b-ad6e-317eca36a388","Type":"ContainerStarted","Data":"7d3bfe5943c945c6b0294561dbcd918ef825816ac6528e08e1c6cbc6da4560ab"}
Aug 04 22:24:29.714639 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:29.714334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:24:29.714939 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:24:29.714920    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5656bdbdc8-4nlxn"
Aug 04 22:25:06.039749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:06.039715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:14.743281 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:14.743231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:25:14.743746 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:14.743406    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 22:25:14.743746 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:14.743480    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs podName:03c5ee73-fd6c-4f21-a662-ec218702b7d9 nodeName:}" failed. No retries permitted until 2026-08-04 22:27:16.74345934 +0000 UTC m=+374.284952512 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs") pod "network-metrics-daemon-bqcgb" (UID: "03c5ee73-fd6c-4f21-a662-ec218702b7d9") : secret "metrics-daemon-secret" not found
Aug 04 22:25:21.534864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.534831    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"]
Aug 04 22:25:21.537640 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.537625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"
Aug 04 22:25:21.539785 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.539764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fz46j\""
Aug 04 22:25:21.541744 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.541725    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67655c4d4d-kzn5x"]
Aug 04 22:25:21.544413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.544396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.547076 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547058    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Aug 04 22:25:21.547252 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 22:25:21.547357 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547334    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Aug 04 22:25:21.547420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547396    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wxr7l\""
Aug 04 22:25:21.547621 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Aug 04 22:25:21.547783 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547762    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 22:25:21.547914 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.547789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Aug 04 22:25:21.550556 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.550539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"]
Aug 04 22:25:21.564174 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.564154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67655c4d4d-kzn5x"]
Aug 04 22:25:21.587313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmk8\" (UniqueName: \"kubernetes.io/projected/6477b4d2-1ebd-41a1-ab4d-7787ed408936-kube-api-access-trmk8\") pod \"network-check-source-65bdbbff45-52mv9\" (UID: \"6477b4d2-1ebd-41a1-ab4d-7787ed408936\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"
Aug 04 22:25:21.587313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-default-certificate\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.587488 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-stats-auth\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.587488 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z798\" (UniqueName: \"kubernetes.io/projected/a520a83c-b775-4e8d-8236-f85692310229-kube-api-access-7z798\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.587488 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.587488 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.587456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.636560 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.636532    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"]
Aug 04 22:25:21.639489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.639464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.640322 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.640305    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wzprd"]
Aug 04 22:25:21.642857 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.642841    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-fj4tc"]
Aug 04 22:25:21.642996 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.642979    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.643860 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.643840    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Aug 04 22:25:21.645715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.644527    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-twzl6\""
Aug 04 22:25:21.645715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.644856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Aug 04 22:25:21.645715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.645621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:25:21.645929 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.645834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.647197 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.647174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Aug 04 22:25:21.647284 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.647176    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 22:25:21.647703 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.647683    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 22:25:21.647947 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.647708    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-25ljd\""
Aug 04 22:25:21.648025 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.648007    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 22:25:21.648498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.648477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 22:25:21.649344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.648980    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Aug 04 22:25:21.650447 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.650421    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 22:25:21.650593 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.650571    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 22:25:21.650701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.650632    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Aug 04 22:25:21.650701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.650572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Aug 04 22:25:21.651138 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.651123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dr7kl\""
Aug 04 22:25:21.651913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.651876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.653980 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.653872    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 22:25:21.654139 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.654125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 22:25:21.654228 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.654206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 22:25:21.654330 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.654316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7b97\""
Aug 04 22:25:21.654653 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.654633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Aug 04 22:25:21.655781 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.655764    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 22:25:21.659027 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.659009    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"]
Aug 04 22:25:21.660177 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.660159    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 22:25:21.662573 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.662553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-fj4tc"]
Aug 04 22:25:21.664438 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.664416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wzprd"]
Aug 04 22:25:21.665555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.665458    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:25:21.688011 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.687988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4zv\" (UniqueName: \"kubernetes.io/projected/56f5848b-6584-4442-b731-8d93865c3a11-kube-api-access-8n4zv\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.688081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-stats-auth\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.688081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-snapshots\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.688165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-trusted-ca\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.688165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-trusted-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688282 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f5848b-6584-4442-b731-8d93865c3a11-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.688282 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.688220    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:22.188205008 +0000 UTC m=+259.729698160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : configmap references non-existent config key: service-ca.crt
Aug 04 22:25:21.688282 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6spf\" (UniqueName: \"kubernetes.io/projected/14a577bd-ca9a-4993-b06c-b5013929952f-kube-api-access-m6spf\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.688282 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-config\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-service-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmk8\" (UniqueName: \"kubernetes.io/projected/6477b4d2-1ebd-41a1-ab4d-7787ed408936-kube-api-access-trmk8\") pod \"network-check-source-65bdbbff45-52mv9\" (UID: \"6477b4d2-1ebd-41a1-ab4d-7787ed408936\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-default-certificate\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrjj\" (UniqueName: \"kubernetes.io/projected/eb8717d6-9bab-47ff-be87-16266aa3e79d-kube-api-access-fmrjj\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56f5848b-6584-4442-b731-8d93865c3a11-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z798\" (UniqueName: \"kubernetes.io/projected/a520a83c-b775-4e8d-8236-f85692310229-kube-api-access-7z798\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.688469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a577bd-ca9a-4993-b06c-b5013929952f-serving-cert\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.688726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8717d6-9bab-47ff-be87-16266aa3e79d-serving-cert\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.688726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.688600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-tmp\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.688726 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.688628    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 22:25:21.688726 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.688682    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:22.188666333 +0000 UTC m=+259.730159485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : secret "router-metrics-certs-default" not found
Aug 04 22:25:21.690508 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.690482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-stats-auth\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.690700 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.690684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-default-certificate\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.699270 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.699251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmk8\" (UniqueName: \"kubernetes.io/projected/6477b4d2-1ebd-41a1-ab4d-7787ed408936-kube-api-access-trmk8\") pod \"network-check-source-65bdbbff45-52mv9\" (UID: \"6477b4d2-1ebd-41a1-ab4d-7787ed408936\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"
Aug 04 22:25:21.700212 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.700192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z798\" (UniqueName: \"kubernetes.io/projected/a520a83c-b775-4e8d-8236-f85692310229-kube-api-access-7z798\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:21.783601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.783576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:21.789661 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4zv\" (UniqueName: \"kubernetes.io/projected/56f5848b-6584-4442-b731-8d93865c3a11-kube-api-access-8n4zv\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.789661 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.789773 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.789773 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-snapshots\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.789773 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.789918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-trusted-ca\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.789918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-trusted-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.789918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.789918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f5848b-6584-4442-b731-8d93865c3a11-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6spf\" (UniqueName: \"kubernetes.io/projected/14a577bd-ca9a-4993-b06c-b5013929952f-kube-api-access-m6spf\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-config\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-service-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.789992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrjj\" (UniqueName: \"kubernetes.io/projected/eb8717d6-9bab-47ff-be87-16266aa3e79d-kube-api-access-fmrjj\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzz9\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56f5848b-6584-4442-b731-8d93865c3a11-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a577bd-ca9a-4993-b06c-b5013929952f-serving-cert\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8717d6-9bab-47ff-be87-16266aa3e79d-serving-cert\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.790128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-tmp\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-snapshots\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56f5848b-6584-4442-b731-8d93865c3a11-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.790654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8717d6-9bab-47ff-be87-16266aa3e79d-tmp\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.791010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790666    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-config\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.791010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/14a577bd-ca9a-4993-b06c-b5013929952f-trusted-ca\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.791010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.790948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-trusted-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.791369 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.791348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8717d6-9bab-47ff-be87-16266aa3e79d-service-ca-bundle\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.792473 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.792451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56f5848b-6584-4442-b731-8d93865c3a11-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.792608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.792592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a577bd-ca9a-4993-b06c-b5013929952f-serving-cert\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.792749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.792735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8717d6-9bab-47ff-be87-16266aa3e79d-serving-cert\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.796614 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.796595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6spf\" (UniqueName: \"kubernetes.io/projected/14a577bd-ca9a-4993-b06c-b5013929952f-kube-api-access-m6spf\") pod \"console-operator-5ccd6b445f-wzprd\" (UID: \"14a577bd-ca9a-4993-b06c-b5013929952f\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.796698 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.796681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4zv\" (UniqueName: \"kubernetes.io/projected/56f5848b-6584-4442-b731-8d93865c3a11-kube-api-access-8n4zv\") pod \"kube-storage-version-migrator-operator-7878f4b88c-8jc8k\" (UID: \"56f5848b-6584-4442-b731-8d93865c3a11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.797185 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.797166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrjj\" (UniqueName: \"kubernetes.io/projected/eb8717d6-9bab-47ff-be87-16266aa3e79d-kube-api-access-fmrjj\") pod \"insights-operator-77696685b-fj4tc\" (UID: \"eb8717d6-9bab-47ff-be87-16266aa3e79d\") " pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.838300 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.838275    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:21.846433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.846415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"
Aug 04 22:25:21.891278 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891278 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzz9\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.891887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.891604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.893095 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.892436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.893095 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.892772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.893095 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.892964    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 22:25:21.893095 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.892983    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f769fc685-cs6qv: secret "image-registry-tls" not found
Aug 04 22:25:21.893514 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:21.893120    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls podName:e8ec3d1d-5694-4598-9ec9-dd3c18748148 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:22.393099147 +0000 UTC m=+259.934592315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls") pod "image-registry-f769fc685-cs6qv" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148") : secret "image-registry-tls" not found
Aug 04 22:25:21.893514 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.893176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.894768 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.894723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.895084 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.895066    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.899531 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.899498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzz9\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.900475 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.900447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:21.953275 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.953238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"
Aug 04 22:25:21.960995 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.960972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:21.967185 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.967167    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9"]
Aug 04 22:25:21.968728 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:21.968699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-fj4tc"
Aug 04 22:25:21.970818 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:21.970793    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6477b4d2_1ebd_41a1_ab4d_7787ed408936.slice/crio-c3d8b781483b33c68f578648cb14ba69e092b5b6ed8cad1784a47614557ff04c WatchSource:0}: Error finding container c3d8b781483b33c68f578648cb14ba69e092b5b6ed8cad1784a47614557ff04c: Status 404 returned error can't find the container with id c3d8b781483b33c68f578648cb14ba69e092b5b6ed8cad1784a47614557ff04c
Aug 04 22:25:22.098625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.098548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k"]
Aug 04 22:25:22.101733 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:22.101702    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f5848b_6584_4442_b731_8d93865c3a11.slice/crio-fc458e0c5eccf1f3c4b3fdfcaf0264ef41db22a71ee17175eff0b49352b26659 WatchSource:0}: Error finding container fc458e0c5eccf1f3c4b3fdfcaf0264ef41db22a71ee17175eff0b49352b26659: Status 404 returned error can't find the container with id fc458e0c5eccf1f3c4b3fdfcaf0264ef41db22a71ee17175eff0b49352b26659
Aug 04 22:25:22.118406 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.118371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wzprd"]
Aug 04 22:25:22.121291 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:22.121264    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a577bd_ca9a_4993_b06c_b5013929952f.slice/crio-a5c5764e74b2a5856fb28cc4920f0cf00b2879f5325b5b55cd699d4fdd0e38a4 WatchSource:0}: Error finding container a5c5764e74b2a5856fb28cc4920f0cf00b2879f5325b5b55cd699d4fdd0e38a4: Status 404 returned error can't find the container with id a5c5764e74b2a5856fb28cc4920f0cf00b2879f5325b5b55cd699d4fdd0e38a4
Aug 04 22:25:22.133839 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.133815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-fj4tc"]
Aug 04 22:25:22.137981 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:22.137958    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8717d6_9bab_47ff_be87_16266aa3e79d.slice/crio-14af3501e31e189a298beef3782f727c06883cb82892348449ef5f4122ca554e WatchSource:0}: Error finding container 14af3501e31e189a298beef3782f727c06883cb82892348449ef5f4122ca554e: Status 404 returned error can't find the container with id 14af3501e31e189a298beef3782f727c06883cb82892348449ef5f4122ca554e
Aug 04 22:25:22.194825 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.194797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:22.194946 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.194861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:22.194998 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.194957    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:23.194935312 +0000 UTC m=+260.736428489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : configmap references non-existent config key: service-ca.crt
Aug 04 22:25:22.194998 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.194962    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 22:25:22.195082 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.195004    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:23.194992295 +0000 UTC m=+260.736485447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : secret "router-metrics-certs-default" not found
Aug 04 22:25:22.396248 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.396217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:22.396393 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.396368    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 22:25:22.396393 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.396383    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f769fc685-cs6qv: secret "image-registry-tls" not found
Aug 04 22:25:22.396463 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:22.396437    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls podName:e8ec3d1d-5694-4598-9ec9-dd3c18748148 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:23.39642045 +0000 UTC m=+260.937913603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls") pod "image-registry-f769fc685-cs6qv" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148") : secret "image-registry-tls" not found
Aug 04 22:25:22.848332 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.848252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fj4tc" event={"ID":"eb8717d6-9bab-47ff-be87-16266aa3e79d","Type":"ContainerStarted","Data":"14af3501e31e189a298beef3782f727c06883cb82892348449ef5f4122ca554e"}
Aug 04 22:25:22.850081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.850044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" event={"ID":"14a577bd-ca9a-4993-b06c-b5013929952f","Type":"ContainerStarted","Data":"a5c5764e74b2a5856fb28cc4920f0cf00b2879f5325b5b55cd699d4fdd0e38a4"}
Aug 04 22:25:22.851422 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.851349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k" event={"ID":"56f5848b-6584-4442-b731-8d93865c3a11","Type":"ContainerStarted","Data":"fc458e0c5eccf1f3c4b3fdfcaf0264ef41db22a71ee17175eff0b49352b26659"}
Aug 04 22:25:22.854496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.854409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9" event={"ID":"6477b4d2-1ebd-41a1-ab4d-7787ed408936","Type":"ContainerStarted","Data":"d422823e6c08e4ad1da1659a53ab16338ba9d461f87871eda9423f2452403aaa"}
Aug 04 22:25:22.854496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.854457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9" event={"ID":"6477b4d2-1ebd-41a1-ab4d-7787ed408936","Type":"ContainerStarted","Data":"c3d8b781483b33c68f578648cb14ba69e092b5b6ed8cad1784a47614557ff04c"}
Aug 04 22:25:22.870601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:22.870375    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-52mv9" podStartSLOduration=1.8703559250000001 podStartE2EDuration="1.870355925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:22.86984194 +0000 UTC m=+260.411335115" watchObservedRunningTime="2026-08-04 22:25:22.870355925 +0000 UTC m=+260.411849102"
Aug 04 22:25:23.203025 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:23.202976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:23.203207 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:23.203089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:23.203350 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.203316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:25.203294886 +0000 UTC m=+262.744788055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : configmap references non-existent config key: service-ca.crt
Aug 04 22:25:23.203350 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.203317    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 22:25:23.203504 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.203375    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:25.203365813 +0000 UTC m=+262.744858966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : secret "router-metrics-certs-default" not found
Aug 04 22:25:23.405063 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:23.404872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:23.405063 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.405016    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 22:25:23.405063 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.405032    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f769fc685-cs6qv: secret "image-registry-tls" not found
Aug 04 22:25:23.405328 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:23.405086    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls podName:e8ec3d1d-5694-4598-9ec9-dd3c18748148 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:25.40507011 +0000 UTC m=+262.946563262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls") pod "image-registry-f769fc685-cs6qv" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148") : secret "image-registry-tls" not found
Aug 04 22:25:24.413746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:24.413688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:24.474912 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:24.474854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:25.220935 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.220905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:25.221053 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.221005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:25.221109 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.221078    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:29.221051528 +0000 UTC m=+266.762544686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : configmap references non-existent config key: service-ca.crt
Aug 04 22:25:25.221172 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.221129    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 22:25:25.221219 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.221188    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:29.221171017 +0000 UTC m=+266.762664170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : secret "router-metrics-certs-default" not found
Aug 04 22:25:25.422936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.422889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:25.423278 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.423053    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 22:25:25.423278 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.423074    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f769fc685-cs6qv: secret "image-registry-tls" not found
Aug 04 22:25:25.423278 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:25.423136    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls podName:e8ec3d1d-5694-4598-9ec9-dd3c18748148 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:29.423117869 +0000 UTC m=+266.964611024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls") pod "image-registry-f769fc685-cs6qv" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148") : secret "image-registry-tls" not found
Aug 04 22:25:25.866547 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.866519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fj4tc" event={"ID":"eb8717d6-9bab-47ff-be87-16266aa3e79d","Type":"ContainerStarted","Data":"1db73ca072085ecf1f845e242d3633f6fe751730092d285f9a13b58af13258bc"}
Aug 04 22:25:25.868073 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.868051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/0.log"
Aug 04 22:25:25.868208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.868084    2582 generic.go:356] "Generic (PLEG): container finished" podID="14a577bd-ca9a-4993-b06c-b5013929952f" containerID="3e58f3fdf72e2af75ed76097f3f780bc205b8d9e6d3b33fb04c5d1aa99820606" exitCode=255
Aug 04 22:25:25.868208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.868143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" event={"ID":"14a577bd-ca9a-4993-b06c-b5013929952f","Type":"ContainerDied","Data":"3e58f3fdf72e2af75ed76097f3f780bc205b8d9e6d3b33fb04c5d1aa99820606"}
Aug 04 22:25:25.868366 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.868346    2582 scope.go:117] "RemoveContainer" containerID="3e58f3fdf72e2af75ed76097f3f780bc205b8d9e6d3b33fb04c5d1aa99820606"
Aug 04 22:25:25.869701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.869675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k" event={"ID":"56f5848b-6584-4442-b731-8d93865c3a11","Type":"ContainerStarted","Data":"7cba3891011e28f918ed0b8acf91360eb5902598b8bbe51784bda5e06980634b"}
Aug 04 22:25:25.885692 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.885641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-fj4tc" podStartSLOduration=1.9625059249999999 podStartE2EDuration="4.885623761s" totalImagesPullingTime="2.923117836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:25.883571758 +0000 UTC m=+263.425064934" watchObservedRunningTime="2026-08-04 22:25:25.885623761 +0000 UTC m=+263.427116917"
Aug 04 22:25:25.913701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:25.913623    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k" podStartSLOduration=1.9575757280000001 podStartE2EDuration="4.913604025s" totalImagesPullingTime="2.956028297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:25.912960773 +0000 UTC m=+263.454453947" watchObservedRunningTime="2026-08-04 22:25:25.913604025 +0000 UTC m=+263.455097200"
Aug 04 22:25:26.109448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.109415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.205300 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.205219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.304790 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.304758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.405173 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.405140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.655387 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.655353    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ngflx"]
Aug 04 22:25:26.658382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.658367    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.660300 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.660275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbwrp\""
Aug 04 22:25:26.660409 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.660386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 22:25:26.660523 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.660506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 22:25:26.666461 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.666437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.669822 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.669801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngflx"]
Aug 04 22:25:26.719991 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.719961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.733815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.733792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.733919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.733835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724525b-ad26-4e95-91f3-4687479a9a5a-data-volume\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.733919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.733856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724525b-ad26-4e95-91f3-4687479a9a5a-crio-socket\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.734001 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.733932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.734001 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.733993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6q7\" (UniqueName: \"kubernetes.io/projected/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-api-access-gj6q7\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.805100 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.805073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:26.835261 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835328 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6q7\" (UniqueName: \"kubernetes.io/projected/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-api-access-gj6q7\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835380 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835380 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:26.835366    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 22:25:26.835457 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724525b-ad26-4e95-91f3-4687479a9a5a-data-volume\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835457 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:26.835420    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:27.335402271 +0000 UTC m=+264.876895429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : configmap "kube-rbac-proxy" not found
Aug 04 22:25:26.835546 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:26.835464    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:26.835546 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724525b-ad26-4e95-91f3-4687479a9a5a-crio-socket\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835546 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:26.835517    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:27.335501075 +0000 UTC m=+264.876994246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:26.835657 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724525b-ad26-4e95-91f3-4687479a9a5a-crio-socket\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.835710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.835695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724525b-ad26-4e95-91f3-4687479a9a5a-data-volume\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.843120 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.843097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6q7\" (UniqueName: \"kubernetes.io/projected/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-api-access-gj6q7\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:26.874083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874062    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/1.log"
Aug 04 22:25:26.874448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874433    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/0.log"
Aug 04 22:25:26.874496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874466    2582 generic.go:356] "Generic (PLEG): container finished" podID="14a577bd-ca9a-4993-b06c-b5013929952f" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75" exitCode=255
Aug 04 22:25:26.874545 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" event={"ID":"14a577bd-ca9a-4993-b06c-b5013929952f","Type":"ContainerDied","Data":"dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"}
Aug 04 22:25:26.874590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874580    2582 scope.go:117] "RemoveContainer" containerID="3e58f3fdf72e2af75ed76097f3f780bc205b8d9e6d3b33fb04c5d1aa99820606"
Aug 04 22:25:26.874810 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.874795    2582 scope.go:117] "RemoveContainer" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"
Aug 04 22:25:26.875024 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:26.875001    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:25:26.892427 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:26.892406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:27.339608 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:27.339562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:27.339823 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:27.339636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:27.339823 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:27.339696    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 22:25:27.339823 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:27.339748    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:27.339823 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:27.339781    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:28.339759858 +0000 UTC m=+265.881253010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : configmap "kube-rbac-proxy" not found
Aug 04 22:25:27.339823 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:27.339799    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:28.339790858 +0000 UTC m=+265.881284015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:27.878790 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:27.878759    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/1.log"
Aug 04 22:25:27.879167 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:27.879083    2582 scope.go:117] "RemoveContainer" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"
Aug 04 22:25:27.879253 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:27.879233    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:25:28.347976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:28.347874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:28.347976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:28.347949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:28.348154 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:28.348011    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 22:25:28.348154 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:28.348062    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:28.348154 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:28.348087    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:30.348070255 +0000 UTC m=+267.889563415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : configmap "kube-rbac-proxy" not found
Aug 04 22:25:28.348154 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:28.348106    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:30.348094661 +0000 UTC m=+267.889587813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:28.440872 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:28.440839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:28.494878 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:28.494848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:29.256720 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:29.256685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:29.257120 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:29.256742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:29.257120 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.256823    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 22:25:29.257120 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.256871    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:37.256857395 +0000 UTC m=+274.798350546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : configmap references non-existent config key: service-ca.crt
Aug 04 22:25:29.257120 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.256885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs podName:a520a83c-b775-4e8d-8236-f85692310229 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:37.256878315 +0000 UTC m=+274.798371466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs") pod "router-default-67655c4d4d-kzn5x" (UID: "a520a83c-b775-4e8d-8236-f85692310229") : secret "router-metrics-certs-default" not found
Aug 04 22:25:29.295166 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:29.295136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:29.458280 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:29.458255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:29.458413 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.458398    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 22:25:29.458457 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.458415    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f769fc685-cs6qv: secret "image-registry-tls" not found
Aug 04 22:25:29.458491 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:29.458465    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls podName:e8ec3d1d-5694-4598-9ec9-dd3c18748148 nodeName:}" failed. No retries permitted until 2026-08-04 22:25:37.45844847 +0000 UTC m=+274.999941624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls") pod "image-registry-f769fc685-cs6qv" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148") : secret "image-registry-tls" not found
Aug 04 22:25:29.620366 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:29.620341    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsjtt_373d37ba-f50e-4fc6-bf05-56f1c09dce5f/dns-node-resolver/0.log"
Aug 04 22:25:30.219865 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.219839    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4z68c_0321c2f7-349d-41ee-844b-7710cdab0e8c/node-ca/0.log"
Aug 04 22:25:30.366110 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.366080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:30.366417 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.366136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:30.366417 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:30.366190    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 22:25:30.366417 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:30.366244    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:34.366230347 +0000 UTC m=+271.907723498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : configmap "kube-rbac-proxy" not found
Aug 04 22:25:30.366417 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:30.366248    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:30.366417 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:30.366284    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:34.366273562 +0000 UTC m=+271.907766713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:30.548815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.548749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:30.651011 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.650974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:30.751111 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.751080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:30.810530 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.810465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:30.959930 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:30.959877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:31.220389 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:31.220353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqhwj_8fbf8a0c-ad84-445a-beab-7a2f37afd2a7/serve-healthcheck-canary/0.log"
Aug 04 22:25:31.450784 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:31.450742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:31.961297 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:31.961266    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:31.961461 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:31.961315    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:31.961630 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:31.961618    2582 scope.go:117] "RemoveContainer" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"
Aug 04 22:25:31.961780 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:31.961764    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:25:32.022245 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:32.022218    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-8jc8k_56f5848b-6584-4442-b731-8d93865c3a11/kube-storage-version-migrator-operator/0.log"
Aug 04 22:25:32.026747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:32.026719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:32.127875 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:32.127844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:32.651563 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:32.651522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:32.841267 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:32.841232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:33.040792 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:33.040718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:33.449082 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:33.449047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.249579 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.249547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.303287 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.303258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.402981 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.402953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:34.403140 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.403049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:34.403140 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:34.403112    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:34.403140 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:34.403124    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 22:25:34.403255 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:34.403180    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:42.403161472 +0000 UTC m=+279.944654624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : configmap "kube-rbac-proxy" not found
Aug 04 22:25:34.403255 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:34.403193    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls podName:0724525b-ad26-4e95-91f3-4687479a9a5a nodeName:}" failed. No retries permitted until 2026-08-04 22:25:42.403187699 +0000 UTC m=+279.944680850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngflx" (UID: "0724525b-ad26-4e95-91f3-4687479a9a5a") : secret "insights-runtime-extractor-tls" not found
Aug 04 22:25:34.442878 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.442846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.544420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.544331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.657237 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.657200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:34.846118 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:34.846037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:35.334087 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:35.334060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:35.860879 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:35.860843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:36.047963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:36.047925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:36.441549 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:36.441511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:36.646660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:36.646624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:36.696049 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:36.695966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:37.088850 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.088771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:37.326216 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.326162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:37.326401 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.326252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:37.326803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.326785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a520a83c-b775-4e8d-8236-f85692310229-service-ca-bundle\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:37.328669 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.328641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a520a83c-b775-4e8d-8236-f85692310229-metrics-certs\") pod \"router-default-67655c4d4d-kzn5x\" (UID: \"a520a83c-b775-4e8d-8236-f85692310229\") " pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:37.454208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.454175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:37.527949 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.527923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:37.530338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.530307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"image-registry-f769fc685-cs6qv\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") " pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:37.574349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.574317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:37.578439 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.578296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67655c4d4d-kzn5x"]
Aug 04 22:25:37.582991 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:37.582968    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda520a83c_b775_4e8d_8236_f85692310229.slice/crio-6d50107f96d94e4e58e0cea26ae5d615f9909f5ee4ca7913bbd75e8c755555cc WatchSource:0}: Error finding container 6d50107f96d94e4e58e0cea26ae5d615f9909f5ee4ca7913bbd75e8c755555cc: Status 404 returned error can't find the container with id 6d50107f96d94e4e58e0cea26ae5d615f9909f5ee4ca7913bbd75e8c755555cc
Aug 04 22:25:37.705339 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.705314    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:25:37.707649 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:37.707621    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ec3d1d_5694_4598_9ec9_dd3c18748148.slice/crio-c7e6c284c62ffc13565a4bdbaff643f427bd394d15ef823a1992dda38e4bf9c3 WatchSource:0}: Error finding container c7e6c284c62ffc13565a4bdbaff643f427bd394d15ef823a1992dda38e4bf9c3: Status 404 returned error can't find the container with id c7e6c284c62ffc13565a4bdbaff643f427bd394d15ef823a1992dda38e4bf9c3
Aug 04 22:25:37.852909 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.852854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:37.911498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.911458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" event={"ID":"e8ec3d1d-5694-4598-9ec9-dd3c18748148","Type":"ContainerStarted","Data":"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"}
Aug 04 22:25:37.911683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.911548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" event={"ID":"e8ec3d1d-5694-4598-9ec9-dd3c18748148","Type":"ContainerStarted","Data":"c7e6c284c62ffc13565a4bdbaff643f427bd394d15ef823a1992dda38e4bf9c3"}
Aug 04 22:25:37.911683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.911574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:25:37.913329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.913305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67655c4d4d-kzn5x" event={"ID":"a520a83c-b775-4e8d-8236-f85692310229","Type":"ContainerStarted","Data":"cad8f9bdc00ca8be7e5ef399aa8a71a7faf831d7276200b0a73d219ae579f84b"}
Aug 04 22:25:37.913412 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.913350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67655c4d4d-kzn5x" event={"ID":"a520a83c-b775-4e8d-8236-f85692310229","Type":"ContainerStarted","Data":"6d50107f96d94e4e58e0cea26ae5d615f9909f5ee4ca7913bbd75e8c755555cc"}
Aug 04 22:25:37.929594 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.929541    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" podStartSLOduration=16.929523516 podStartE2EDuration="16.929523516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:37.928191857 +0000 UTC m=+275.469685044" watchObservedRunningTime="2026-08-04 22:25:37.929523516 +0000 UTC m=+275.471016691"
Aug 04 22:25:37.944056 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:37.944018    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67655c4d4d-kzn5x" podStartSLOduration=16.944004425 podStartE2EDuration="16.944004425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:37.943452536 +0000 UTC m=+275.484945733" watchObservedRunningTime="2026-08-04 22:25:37.944004425 +0000 UTC m=+275.485497599"
Aug 04 22:25:38.045045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.044955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:38.245114 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.245070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:38.455290 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.455257    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:38.457962 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.457941    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:38.644290 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.644257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:38.916607 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.916578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:38.917735 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:38.917714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67655c4d4d-kzn5x"
Aug 04 22:25:39.244269 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:39.244185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:39.345416 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:39.345387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:39.445744 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:39.445716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:40.045656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:40.045624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:40.648146 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:40.648105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:40.846692 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:40.846663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:41.040676 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:41.040602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:41.247279 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:41.247256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:42.467559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.467507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:42.468052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.467594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:42.468052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.468022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724525b-ad26-4e95-91f3-4687479a9a5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:42.470036 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.470014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724525b-ad26-4e95-91f3-4687479a9a5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngflx\" (UID: \"0724525b-ad26-4e95-91f3-4687479a9a5a\") " pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:42.567189 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.567156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngflx"
Aug 04 22:25:42.695699 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.695669    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngflx"]
Aug 04 22:25:42.698557 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:42.698518    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724525b_ad26_4e95_91f3_4687479a9a5a.slice/crio-df381e6efc5e683e36472e1e274750b0d59f8f25ddc560301d721dc5f261bad3 WatchSource:0}: Error finding container df381e6efc5e683e36472e1e274750b0d59f8f25ddc560301d721dc5f261bad3: Status 404 returned error can't find the container with id df381e6efc5e683e36472e1e274750b0d59f8f25ddc560301d721dc5f261bad3
Aug 04 22:25:42.929779 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.929742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngflx" event={"ID":"0724525b-ad26-4e95-91f3-4687479a9a5a","Type":"ContainerStarted","Data":"a91ad388db093395afe7123fb420e349013f4d9db05f761a0c25f4b5c970d3a5"}
Aug 04 22:25:42.929935 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:42.929796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngflx" event={"ID":"0724525b-ad26-4e95-91f3-4687479a9a5a","Type":"ContainerStarted","Data":"df381e6efc5e683e36472e1e274750b0d59f8f25ddc560301d721dc5f261bad3"}
Aug 04 22:25:43.934007 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:43.933975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngflx" event={"ID":"0724525b-ad26-4e95-91f3-4687479a9a5a","Type":"ContainerStarted","Data":"6f6ce085f74ee705f41a7dd6ae0dccaa735924f4ab492880cc37c17a1a1ecc1c"}
Aug 04 22:25:45.603112 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:45.603075    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fggh6" podUID="e68ddc17-b853-463e-9a50-720ea2758229"
Aug 04 22:25:45.942810 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:45.942779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:25:45.942810 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:45.942790    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngflx" event={"ID":"0724525b-ad26-4e95-91f3-4687479a9a5a","Type":"ContainerStarted","Data":"ac2e7ebf217c04c2da4432966c79cf48c05c15191f2f84e8d9d81316bd4f861b"}
Aug 04 22:25:45.959676 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:45.959627    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ngflx" podStartSLOduration=17.751116374 podStartE2EDuration="19.959609133s" totalImagesPullingTime="2.208492759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 22:25:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:45.958712405 +0000 UTC m=+283.500205580" watchObservedRunningTime="2026-08-04 22:25:45.959609133 +0000 UTC m=+283.501102306"
Aug 04 22:25:46.005160 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.005139    2582 scope.go:117] "RemoveContainer" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"
Aug 04 22:25:46.947389 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.947358    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:25:46.947794 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.947756    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/1.log"
Aug 04 22:25:46.947794 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.947785    2582 generic.go:356] "Generic (PLEG): container finished" podID="14a577bd-ca9a-4993-b06c-b5013929952f" containerID="29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887" exitCode=255
Aug 04 22:25:46.947882 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.947863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" event={"ID":"14a577bd-ca9a-4993-b06c-b5013929952f","Type":"ContainerDied","Data":"29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887"}
Aug 04 22:25:46.947951 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.947937    2582 scope.go:117] "RemoveContainer" containerID="dd69c572e65bed1fb4a5f01b4b39f711e44361f94096bb76851990ae02335a75"
Aug 04 22:25:46.948297 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:46.948280    2582 scope.go:117] "RemoveContainer" containerID="29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887"
Aug 04 22:25:46.948458 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:46.948434    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:25:47.114836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:47.114809    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:25:47.254562 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:47.254490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:47.359259 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:47.359222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:47.952344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:47.952317    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:25:48.712628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:48.712592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:25:48.715150 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:48.715114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e68ddc17-b853-463e-9a50-720ea2758229-metrics-tls\") pod \"dns-default-fggh6\" (UID: \"e68ddc17-b853-463e-9a50-720ea2758229\") " pod="openshift-dns/dns-default-fggh6"
Aug 04 22:25:48.946004 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:48.945979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d46fp\""
Aug 04 22:25:48.954443 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:48.954418    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:25:49.077738 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:49.077711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fggh6"]
Aug 04 22:25:49.080722 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:25:49.080699    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ddc17_b853_463e_9a50_720ea2758229.slice/crio-0397c6412621dacb1d965bc2b7469715d4a170f4ef6216600c897889b86df82a WatchSource:0}: Error finding container 0397c6412621dacb1d965bc2b7469715d4a170f4ef6216600c897889b86df82a: Status 404 returned error can't find the container with id 0397c6412621dacb1d965bc2b7469715d4a170f4ef6216600c897889b86df82a
Aug 04 22:25:49.959977 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:49.959939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fggh6" event={"ID":"e68ddc17-b853-463e-9a50-720ea2758229","Type":"ContainerStarted","Data":"0397c6412621dacb1d965bc2b7469715d4a170f4ef6216600c897889b86df82a"}
Aug 04 22:25:50.964510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:50.964427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fggh6" event={"ID":"e68ddc17-b853-463e-9a50-720ea2758229","Type":"ContainerStarted","Data":"8f81f4062e86df72ed4d4cddd5722915f38ddfe7b06e4421f1172fee3620b4a9"}
Aug 04 22:25:50.964510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:50.964487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fggh6" event={"ID":"e68ddc17-b853-463e-9a50-720ea2758229","Type":"ContainerStarted","Data":"292882117ab1b84537e0e9f5b21f081a0c4acbdc8ff40015c974294547713762"}
Aug 04 22:25:50.964991 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:50.964552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:25:50.980776 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:50.980735    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fggh6" podStartSLOduration=251.444654392 podStartE2EDuration="4m12.980722972s" totalImagesPullingTime="1.53606858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:25:50.97942353 +0000 UTC m=+288.520916706" watchObservedRunningTime="2026-08-04 22:25:50.980722972 +0000 UTC m=+288.522216146"
Aug 04 22:25:51.961342 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:51.961312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:51.961482 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:51.961362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:25:51.961652 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:51.961636    2582 scope.go:117] "RemoveContainer" containerID="29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887"
Aug 04 22:25:51.961800 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:51.961782    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:25:52.865442 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:52.865404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:52.965697 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:52.965650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:55.445165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:55.445130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:56.526144 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:56.526106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:57.006095 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:25:57.006058    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bqcgb" podUID="03c5ee73-fd6c-4f21-a662-ec218702b7d9"
Aug 04 22:25:57.120659 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:57.120627    2582 patch_prober.go:28] interesting pod/image-registry-f769fc685-cs6qv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 04 22:25:57.120797 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:57.120673    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 22:25:57.623148 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:57.623115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:57.732031 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:57.731997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:58.165773 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:58.165744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:59.506638 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:59.506607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:59.556592 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:59.556559    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:59.625829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:59.625803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:25:59.726850 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:25:59.726822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:00.628816 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:00.628790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:00.969403 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:00.969328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fggh6"
Aug 04 22:26:02.771876 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.771843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:02.849617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.849592    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:26:02.850435 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.850409    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:26:02.860527 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.860511    2582 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 22:26:02.978083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.978047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:02.982039 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.982015    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"]
Aug 04 22:26:02.988017 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.987988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:02.992178 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992154    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 22:26:02.992178 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992171    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 22:26:02.992312 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992259    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Aug 04 22:26:02.992963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Aug 04 22:26:02.993074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dn4vh\""
Aug 04 22:26:02.993074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.992964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 22:26:02.996072 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:02.996053    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"]
Aug 04 22:26:03.004519 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.004398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae58b47-ec22-4e75-9142-0c59114a3b9e-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.004519 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.004431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.004519 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.004478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.008322 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.004534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvgr\" (UniqueName: \"kubernetes.io/projected/fae58b47-ec22-4e75-9142-0c59114a3b9e-kube-api-access-gfvgr\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.009233 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.009217    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"]
Aug 04 22:26:03.012231 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.012214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.014338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.014320    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 22:26:03.014424 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.014357    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 22:26:03.014424 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.014400    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 22:26:03.014424 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.014414    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gsccc\""
Aug 04 22:26:03.021294 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.021260    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bp984"]
Aug 04 22:26:03.023639 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.023592    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"]
Aug 04 22:26:03.023732 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.023679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.025672 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.025655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 22:26:03.025792 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.025774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 22:26:03.025866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.025809    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hmr7p\""
Aug 04 22:26:03.026110 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.026094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 22:26:03.073473 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.073445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.105793 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.105767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-textfile\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.105887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.105797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5fn\" (UniqueName: \"kubernetes.io/projected/895593f6-4d7d-44d3-acbe-a67910294dcc-kube-api-access-9g5fn\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.105887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.105823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvgr\" (UniqueName: \"kubernetes.io/projected/fae58b47-ec22-4e75-9142-0c59114a3b9e-kube-api-access-gfvgr\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.105887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.105840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-metrics-client-ca\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106018 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.105968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae58b47-ec22-4e75-9142-0c59114a3b9e-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.106018 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106079 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.106079 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-root\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fce7924e-ce59-4806-850f-80cc8d8c00b3-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-sys\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106278 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106278 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106339 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-wtmp\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.106339 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bhd\" (UniqueName: \"kubernetes.io/projected/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-api-access-95bhd\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.106421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.106636 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.106619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae58b47-ec22-4e75-9142-0c59114a3b9e-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.108421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.108399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.108777 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.108760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fae58b47-ec22-4e75-9142-0c59114a3b9e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.115613 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.115587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvgr\" (UniqueName: \"kubernetes.io/projected/fae58b47-ec22-4e75-9142-0c59114a3b9e-kube-api-access-gfvgr\") pod \"openshift-state-metrics-6bb4d7b4dd-twb9m\" (UID: \"fae58b47-ec22-4e75-9142-0c59114a3b9e\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.172659 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.172639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.206811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.206937 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-textfile\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.206937 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5fn\" (UniqueName: \"kubernetes.io/projected/895593f6-4d7d-44d3-acbe-a67910294dcc-kube-api-access-9g5fn\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.206937 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-metrics-client-ca\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.206998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-root\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:26:03.207102    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-root\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-textfile\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:26:03.207168    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls podName:895593f6-4d7d-44d3-acbe-a67910294dcc nodeName:}" failed. No retries permitted until 2026-08-04 22:26:03.707149443 +0000 UTC m=+301.248642599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls") pod "node-exporter-bp984" (UID: "895593f6-4d7d-44d3-acbe-a67910294dcc") : secret "node-exporter-tls" not found
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fce7924e-ce59-4806-850f-80cc8d8c00b3-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-sys\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-wtmp\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-sys\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fce7924e-ce59-4806-850f-80cc8d8c00b3-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-metrics-client-ca\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-wtmp\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bhd\" (UniqueName: \"kubernetes.io/projected/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-api-access-95bhd\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.207838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.207789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.208169 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.208140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.209513 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.209489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.209843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.209829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.209961 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.209941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.217013 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.216993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5fn\" (UniqueName: \"kubernetes.io/projected/895593f6-4d7d-44d3-acbe-a67910294dcc-kube-api-access-9g5fn\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.217409 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.217394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bhd\" (UniqueName: \"kubernetes.io/projected/fce7924e-ce59-4806-850f-80cc8d8c00b3-kube-api-access-95bhd\") pod \"kube-state-metrics-6bd9664c4-rsfct\" (UID: \"fce7924e-ce59-4806-850f-80cc8d8c00b3\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.275190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.275139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.297264 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.297242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"
Aug 04 22:26:03.320962 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.320941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"
Aug 04 22:26:03.374932 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.373537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.431808 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.431783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m"]
Aug 04 22:26:03.434057 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:03.434029    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae58b47_ec22_4e75_9142_0c59114a3b9e.slice/crio-360f832bba8dd05bda4fbf9fc09845c723ee3703bbac7d7826e0e25ce8607ea4 WatchSource:0}: Error finding container 360f832bba8dd05bda4fbf9fc09845c723ee3703bbac7d7826e0e25ce8607ea4: Status 404 returned error can't find the container with id 360f832bba8dd05bda4fbf9fc09845c723ee3703bbac7d7826e0e25ce8607ea4
Aug 04 22:26:03.460807 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.460779    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct"]
Aug 04 22:26:03.465830 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:03.465807    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce7924e_ce59_4806_850f_80cc8d8c00b3.slice/crio-1681f43d4dc253980682ec8db6a5670ef7942885cc7db1d160b0f28b5f844802 WatchSource:0}: Error finding container 1681f43d4dc253980682ec8db6a5670ef7942885cc7db1d160b0f28b5f844802: Status 404 returned error can't find the container with id 1681f43d4dc253980682ec8db6a5670ef7942885cc7db1d160b0f28b5f844802
Aug 04 22:26:03.563726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.563700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.711503 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.711475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.712931 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.712880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:03.714071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.714055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/895593f6-4d7d-44d3-acbe-a67910294dcc-node-exporter-tls\") pod \"node-exporter-bp984\" (UID: \"895593f6-4d7d-44d3-acbe-a67910294dcc\") " pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.932086 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.932061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bp984"
Aug 04 22:26:03.941312 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:03.941287    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895593f6_4d7d_44d3_acbe_a67910294dcc.slice/crio-f690a5315aaeb256d2407800e62d2f9551dbb424cb9c75f713443ad83433a930 WatchSource:0}: Error finding container f690a5315aaeb256d2407800e62d2f9551dbb424cb9c75f713443ad83433a930: Status 404 returned error can't find the container with id f690a5315aaeb256d2407800e62d2f9551dbb424cb9c75f713443ad83433a930
Aug 04 22:26:03.983230 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:03.983186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:04.004386 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.004364    2582 scope.go:117] "RemoveContainer" containerID="29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887"
Aug 04 22:26:04.004527 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:26:04.004511    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wzprd_openshift-console-operator(14a577bd-ca9a-4993-b06c-b5013929952f)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f"
Aug 04 22:26:04.009590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.009565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m" event={"ID":"fae58b47-ec22-4e75-9142-0c59114a3b9e","Type":"ContainerStarted","Data":"9bb8cdc2cd3befe6153eab142ffc60e87142619a7f05ab6a104ce2c941363d3d"}
Aug 04 22:26:04.009684 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.009612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m" event={"ID":"fae58b47-ec22-4e75-9142-0c59114a3b9e","Type":"ContainerStarted","Data":"f3ad736f0f51f8ece7140ecd9324f642437d8e09188b81d0ce46bdc9a7def9e9"}
Aug 04 22:26:04.009684 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.009636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m" event={"ID":"fae58b47-ec22-4e75-9142-0c59114a3b9e","Type":"ContainerStarted","Data":"360f832bba8dd05bda4fbf9fc09845c723ee3703bbac7d7826e0e25ce8607ea4"}
Aug 04 22:26:04.010713 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.010683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bp984" event={"ID":"895593f6-4d7d-44d3-acbe-a67910294dcc","Type":"ContainerStarted","Data":"f690a5315aaeb256d2407800e62d2f9551dbb424cb9c75f713443ad83433a930"}
Aug 04 22:26:04.011704 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.011686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct" event={"ID":"fce7924e-ce59-4806-850f-80cc8d8c00b3","Type":"ContainerStarted","Data":"1681f43d4dc253980682ec8db6a5670ef7942885cc7db1d160b0f28b5f844802"}
Aug 04 22:26:04.061493 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.061470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:04.365054 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:04.364962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:05.949333 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:05.949256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:06.007601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.007568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:06.020990 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.020963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m" event={"ID":"fae58b47-ec22-4e75-9142-0c59114a3b9e","Type":"ContainerStarted","Data":"60fd0c765dcb41063b15a0826acbd1e4e7fd5cedbe0a6aa4a396ae37ae929780"}
Aug 04 22:26:06.022599 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.022576    2582 generic.go:356] "Generic (PLEG): container finished" podID="895593f6-4d7d-44d3-acbe-a67910294dcc" containerID="9ae314ed46bb91212cb4c0a4d86bf8a56612bca09c743d1774742f67812fb71d" exitCode=0
Aug 04 22:26:06.022707 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.022646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bp984" event={"ID":"895593f6-4d7d-44d3-acbe-a67910294dcc","Type":"ContainerDied","Data":"9ae314ed46bb91212cb4c0a4d86bf8a56612bca09c743d1774742f67812fb71d"}
Aug 04 22:26:06.024822 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.024801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct" event={"ID":"fce7924e-ce59-4806-850f-80cc8d8c00b3","Type":"ContainerStarted","Data":"c45f759eeb3d04d2fcf740b99e22133c1cc7ec07d67d05f2f951b2ac494bf32d"}
Aug 04 22:26:06.024942 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.024844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct" event={"ID":"fce7924e-ce59-4806-850f-80cc8d8c00b3","Type":"ContainerStarted","Data":"9b4fd0c01b499912af556cb8745f327c6a5b26701a0c6b6a72e73dacc9c43cf0"}
Aug 04 22:26:06.024942 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.024870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct" event={"ID":"fce7924e-ce59-4806-850f-80cc8d8c00b3","Type":"ContainerStarted","Data":"a2943a7015ad3defb8394fc75821ff20fc97e3b906a101de821258a2b949e10c"}
Aug 04 22:26:06.038104 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.038067    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-twb9m" podStartSLOduration=2.218388652 podStartE2EDuration="4.03805427s" totalImagesPullingTime="1.819665618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:06.037514906 +0000 UTC m=+303.579008081" watchObservedRunningTime="2026-08-04 22:26:06.03805427 +0000 UTC m=+303.579547444"
Aug 04 22:26:06.056409 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.056370    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-rsfct" podStartSLOduration=2.131630537 podStartE2EDuration="4.05635845s" totalImagesPullingTime="1.924727913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:06.05482477 +0000 UTC m=+303.596317955" watchObservedRunningTime="2026-08-04 22:26:06.05635845 +0000 UTC m=+303.597851623"
Aug 04 22:26:06.089139 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.089116    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-578447c689-j4kn8"]
Aug 04 22:26:06.091507 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.091485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.094726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.094591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Aug 04 22:26:06.094726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.094610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w4wmn\""
Aug 04 22:26:06.094726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.094698    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Aug 04 22:26:06.094726 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.094706    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dvugnfgu8ei2m\""
Aug 04 22:26:06.095115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.095097    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Aug 04 22:26:06.095226 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.095125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Aug 04 22:26:06.095226 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.095133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Aug 04 22:26:06.108809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.108791    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578447c689-j4kn8"]
Aug 04 22:26:06.132936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.132879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133118 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.132999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133118 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133247 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48499a1b-4d1a-453b-a4bb-a5360294d0f2-metrics-client-ca\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133446 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133446 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-grpc-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.133847 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.133824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvngd\" (UniqueName: \"kubernetes.io/projected/48499a1b-4d1a-453b-a4bb-a5360294d0f2-kube-api-access-zvngd\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235090 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235090 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-grpc-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvngd\" (UniqueName: \"kubernetes.io/projected/48499a1b-4d1a-453b-a4bb-a5360294d0f2-kube-api-access-zvngd\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.235448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.235380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48499a1b-4d1a-453b-a4bb-a5360294d0f2-metrics-client-ca\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.236131 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.236092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48499a1b-4d1a-453b-a4bb-a5360294d0f2-metrics-client-ca\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.237743 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.237718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.237981 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.237962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.238214 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.238194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.238301 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.238266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.238406 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.238388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-grpc-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.238515 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.238495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48499a1b-4d1a-453b-a4bb-a5360294d0f2-secret-thanos-querier-tls\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.243836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.243800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvngd\" (UniqueName: \"kubernetes.io/projected/48499a1b-4d1a-453b-a4bb-a5360294d0f2-kube-api-access-zvngd\") pod \"thanos-querier-578447c689-j4kn8\" (UID: \"48499a1b-4d1a-453b-a4bb-a5360294d0f2\") " pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.385878 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.385821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:06.401750 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.401734    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:06.536246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.536116    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578447c689-j4kn8"]
Aug 04 22:26:06.538640 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:06.538604    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48499a1b_4d1a_453b_a4bb_a5360294d0f2.slice/crio-5c7c9f529340efe92799b7dac1f24863ff1d993e45e441cc9089cb0d7813eab4 WatchSource:0}: Error finding container 5c7c9f529340efe92799b7dac1f24863ff1d993e45e441cc9089cb0d7813eab4: Status 404 returned error can't find the container with id 5c7c9f529340efe92799b7dac1f24863ff1d993e45e441cc9089cb0d7813eab4
Aug 04 22:26:06.540729 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.540710    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 22:26:06.543580 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:06.543554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:07.029480 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.029448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"5c7c9f529340efe92799b7dac1f24863ff1d993e45e441cc9089cb0d7813eab4"}
Aug 04 22:26:07.031602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.031574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bp984" event={"ID":"895593f6-4d7d-44d3-acbe-a67910294dcc","Type":"ContainerStarted","Data":"75a9e5bf4cd06f12402169ad9599914d8fc741cba659c849a5f4e71eb97aa5bd"}
Aug 04 22:26:07.031712 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.031620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bp984" event={"ID":"895593f6-4d7d-44d3-acbe-a67910294dcc","Type":"ContainerStarted","Data":"f033fe9381d019e4630ed2d3a23e1af60e8783e0e2e8e20be27be525d4444690"}
Aug 04 22:26:07.052439 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.052403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bp984" podStartSLOduration=3.498485574 podStartE2EDuration="5.052389594s" totalImagesPullingTime="1.45085197s" totalInitContainerRuntime="103.05205ms" isStatefulPod=true podCreationTimestamp="2026-08-04 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:07.051793757 +0000 UTC m=+304.593286932" watchObservedRunningTime="2026-08-04 22:26:07.052389594 +0000 UTC m=+304.593882768"
Aug 04 22:26:07.121035 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.121012    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:26:07.765224 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.765194    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"]
Aug 04 22:26:07.767308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.767286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.767880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.767853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:07.769461 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.769446    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Aug 04 22:26:07.769611 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.769597    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mr8pk\""
Aug 04 22:26:07.781146 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.781122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"]
Aug 04 22:26:07.850466 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.850439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da3b55d0-6766-4865-afeb-7241dd4a5e88-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.850574 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.850505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsg72\" (UniqueName: \"kubernetes.io/projected/da3b55d0-6766-4865-afeb-7241dd4a5e88-kube-api-access-rsg72\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.951778 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.951749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da3b55d0-6766-4865-afeb-7241dd4a5e88-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.951888 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.951839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsg72\" (UniqueName: \"kubernetes.io/projected/da3b55d0-6766-4865-afeb-7241dd4a5e88-kube-api-access-rsg72\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.954403 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.954372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da3b55d0-6766-4865-afeb-7241dd4a5e88-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.964368 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.964346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsg72\" (UniqueName: \"kubernetes.io/projected/da3b55d0-6766-4865-afeb-7241dd4a5e88-kube-api-access-rsg72\") pod \"monitoring-plugin-6dffbbd67-6rgg2\" (UID: \"da3b55d0-6766-4865-afeb-7241dd4a5e88\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:07.997651 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:07.997625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:08.076629 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.076520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:08.170393 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.170303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:08.213434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.213252    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"]
Aug 04 22:26:08.216329 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:08.216289    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3b55d0_6766_4865_afeb_7241dd4a5e88.slice/crio-fb7fbd4905a18da7b9385e0dcad1ec1b2720304ebed3a819093b2af4d606605d WatchSource:0}: Error finding container fb7fbd4905a18da7b9385e0dcad1ec1b2720304ebed3a819093b2af4d606605d: Status 404 returned error can't find the container with id fb7fbd4905a18da7b9385e0dcad1ec1b2720304ebed3a819093b2af4d606605d
Aug 04 22:26:08.269959 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.269930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:08.519021 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.518981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:08.620329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:08.620296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:09.043413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.043371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"4c46b7498e640269b5f2f09096081bd517b2877797a0ae024f3f68d9666147ea"}
Aug 04 22:26:09.043554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.043459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"aedcdaef826dd84ab2bfb4f078030019edd6c6278abce6bfc70e0657501de4d5"}
Aug 04 22:26:09.043554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.043498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"70b48ce84ed2f5c5b1d3e87b136385a10dc65cdb772e136c35b809e5fccc1ace"}
Aug 04 22:26:09.044820 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.044795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2" event={"ID":"da3b55d0-6766-4865-afeb-7241dd4a5e88","Type":"ContainerStarted","Data":"fb7fbd4905a18da7b9385e0dcad1ec1b2720304ebed3a819093b2af4d606605d"}
Aug 04 22:26:09.249332 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.249298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:09.257516 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.257493    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:26:09.260977 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.260961    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.264205 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.264184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 22:26:09.264316 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.264245    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 22:26:09.265284 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.265263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 22:26:09.265631 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.265612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 22:26:09.266150 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266128    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 22:26:09.266454 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266334    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 22:26:09.266549 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266503    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 22:26:09.266644 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266619    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 22:26:09.267194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266943    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 22:26:09.267194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.266950    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 22:26:09.267194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 22:26:09.267194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267110    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 22:26:09.267454 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267303    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 22:26:09.267543 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267524    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 22:26:09.267641 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267622    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tpprg\""
Aug 04 22:26:09.267781 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.267764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-de1s36g4auta8\""
Aug 04 22:26:09.269107 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.269085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 22:26:09.279839 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.279816    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:26:09.364134 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364267 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364267 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364343 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364343 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7vw\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364343 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.364770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.364653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466196 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466537 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466537 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466537 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7vw\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.466811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467059 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467059 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467059 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467059 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.466932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467434 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.467398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.467889 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.467855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.468736 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.468710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.468949 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.468874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.471014 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.470994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.471688 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.471667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.475374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.475154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.475374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.475323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.475374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.475354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.475576 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.475398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.476364 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.476322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.476451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.476408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.476451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.476409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.476795 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.476777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.477219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.477163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.477219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.477191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.477395 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.477234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.478811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.478772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.479870 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.479835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7vw\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw\") pod \"prometheus-k8s-0\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.573534 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.573461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:09.787303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.787266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:09.888382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.888234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:09.987971 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:09.987940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:10.006188 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.006121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:26:10.036579 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.036552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:26:10.040055 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:26:10.040023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa3ce60_50bd_46e7_b822_0468b2b4e66d.slice/crio-ae8bf19a01bd58e3c6895e5432d3330a84e20f4d88e9719284e1ddbbb4bf2349 WatchSource:0}: Error finding container ae8bf19a01bd58e3c6895e5432d3330a84e20f4d88e9719284e1ddbbb4bf2349: Status 404 returned error can't find the container with id ae8bf19a01bd58e3c6895e5432d3330a84e20f4d88e9719284e1ddbbb4bf2349
Aug 04 22:26:10.052782 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.052728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"7443da57c52f7464740c78b4f671dabb480fbde85c0a8b3970030d387f474240"}
Aug 04 22:26:10.052889 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.052800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"90ff27e1845c561341f50113678a60cc74bb1a38e39e5cb8477af4e12b658a92"}
Aug 04 22:26:10.054469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.054440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2" event={"ID":"da3b55d0-6766-4865-afeb-7241dd4a5e88","Type":"ContainerStarted","Data":"f99f57fea3b93e6267c574a5594438abdacbd7b2a68e3ae6eac3edde0d712fac"}
Aug 04 22:26:10.054724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.054676    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:10.055986 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.055945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"ae8bf19a01bd58e3c6895e5432d3330a84e20f4d88e9719284e1ddbbb4bf2349"}
Aug 04 22:26:10.059733 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.059713    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2"
Aug 04 22:26:10.069344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:10.069298    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-6rgg2" podStartSLOduration=1.401226778 podStartE2EDuration="3.069280686s" totalImagesPullingTime="1.668053908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:10.06807896 +0000 UTC m=+307.609572138" watchObservedRunningTime="2026-08-04 22:26:10.069280686 +0000 UTC m=+307.610773861"
Aug 04 22:26:11.062714 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:11.062689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" event={"ID":"48499a1b-4d1a-453b-a4bb-a5360294d0f2","Type":"ContainerStarted","Data":"a7e5dd8fb3eff4c1ccade41ae65de044f0492b835929df35ed1ec309290ad413"}
Aug 04 22:26:11.063050 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:11.062815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:11.064338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:11.064315    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251" exitCode=0
Aug 04 22:26:11.064429 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:11.064407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"}
Aug 04 22:26:11.083789 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:11.083725    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8" podStartSLOduration=1.930321051 podStartE2EDuration="5.083712891s" totalImagesPullingTime="3.15339184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:26:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:11.082200216 +0000 UTC m=+308.623693383" watchObservedRunningTime="2026-08-04 22:26:11.083712891 +0000 UTC m=+308.625206066"
Aug 04 22:26:12.134809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.134756    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerName="registry" containerID="cri-o://d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114" gracePeriod=30
Aug 04 22:26:12.405859 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.405828    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:26:12.494717 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.494683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.494919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.494747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.494919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.494769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495056 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495017    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495132 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495063    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rzz9\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495132 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495248 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495137    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495248 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:26:12.495248 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca\") pod \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\" (UID: \"e8ec3d1d-5694-4598-9ec9-dd3c18748148\") "
Aug 04 22:26:12.495611 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.495562    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-certificates\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.496131 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.496102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:26:12.498705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.498672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:26:12.498705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.498686    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:26:12.499028 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.499002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:26:12.499028 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.499019    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:26:12.499146 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.499036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9" (OuterVolumeSpecName: "kube-api-access-6rzz9") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "kube-api-access-6rzz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:26:12.505536 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.505515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e8ec3d1d-5694-4598-9ec9-dd3c18748148" (UID: "e8ec3d1d-5694-4598-9ec9-dd3c18748148"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:26:12.532758 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.532662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:12.588176 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.588041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:12.596970 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.596925    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-registry-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.596970 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.596951    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8ec3d1d-5694-4598-9ec9-dd3c18748148-ca-trust-extracted\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.596970 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.596968    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-image-registry-private-configuration\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.597198 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.596982    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rzz9\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-kube-api-access-6rzz9\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.597198 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.596995    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8ec3d1d-5694-4598-9ec9-dd3c18748148-installation-pull-secrets\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.597198 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.597008    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8ec3d1d-5694-4598-9ec9-dd3c18748148-bound-sa-token\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:12.597198 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:12.597019    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8ec3d1d-5694-4598-9ec9-dd3c18748148-trusted-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:26:13.074092 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.074061    2582 generic.go:356] "Generic (PLEG): container finished" podID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerID="d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114" exitCode=0
Aug 04 22:26:13.074222 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.074113    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f769fc685-cs6qv"
Aug 04 22:26:13.074222 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.074156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" event={"ID":"e8ec3d1d-5694-4598-9ec9-dd3c18748148","Type":"ContainerDied","Data":"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"}
Aug 04 22:26:13.074222 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.074215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f769fc685-cs6qv" event={"ID":"e8ec3d1d-5694-4598-9ec9-dd3c18748148","Type":"ContainerDied","Data":"c7e6c284c62ffc13565a4bdbaff643f427bd394d15ef823a1992dda38e4bf9c3"}
Aug 04 22:26:13.074333 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.074238    2582 scope.go:117] "RemoveContainer" containerID="d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"
Aug 04 22:26:13.088584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.088562    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:26:13.092926 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:13.092888    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f769fc685-cs6qv"]
Aug 04 22:26:14.123771 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:14.123555    2582 scope.go:117] "RemoveContainer" containerID="d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"
Aug 04 22:26:14.124063 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:26:14.123910    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114\": container with ID starting with d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114 not found: ID does not exist" containerID="d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"
Aug 04 22:26:14.124063 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:14.123942    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114"} err="failed to get container status \"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114\": rpc error: code = NotFound desc = could not find container \"d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114\": container with ID starting with d5634287925c8b5290e438b27aba6e15b61bbe0e810e9b1cf373c0c44ce8c114 not found: ID does not exist"
Aug 04 22:26:15.009238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.009207    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" path="/var/lib/kubelet/pods/e8ec3d1d-5694-4598-9ec9-dd3c18748148/volumes"
Aug 04 22:26:15.085939 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.085912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"}
Aug 04 22:26:15.086067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.085968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"}
Aug 04 22:26:15.086067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.086005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"}
Aug 04 22:26:15.086067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.086035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"}
Aug 04 22:26:15.086067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.086057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"}
Aug 04 22:26:15.086196 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.086079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerStarted","Data":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"}
Aug 04 22:26:15.119465 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:15.119416    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.083768306 podStartE2EDuration="6.119400261s" totalImagesPullingTime="4.012710754s" totalInitContainerRuntime="22.921201ms" isStatefulPod=false podCreationTimestamp="2026-08-04 22:26:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:15.117693484 +0000 UTC m=+312.659186657" watchObservedRunningTime="2026-08-04 22:26:15.119400261 +0000 UTC m=+312.660893472"
Aug 04 22:26:17.075055 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:17.075027    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-578447c689-j4kn8"
Aug 04 22:26:17.507980 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:17.507947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:17.558294 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:17.558262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:18.004711 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.004684    2582 scope.go:117] "RemoveContainer" containerID="29e481a44afb4ec3b8e1d15eb91de0e6aba18a34c1341ebd09857a081d161887"
Aug 04 22:26:18.100313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.100291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:26:18.100675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.100345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" event={"ID":"14a577bd-ca9a-4993-b06c-b5013929952f","Type":"ContainerStarted","Data":"2014f37bf46ea444fcd5fb68c81337f8e9028975fb5a9abb3b0723cf9426cfcb"}
Aug 04 22:26:18.100675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.100619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:26:18.102300 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.102277    2582 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-wzprd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.12:8443/readyz\": dial tcp 10.134.0.12:8443: connect: connection refused" start-of-body=
Aug 04 22:26:18.102420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.102323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podUID="14a577bd-ca9a-4993-b06c-b5013929952f" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.12:8443/readyz\": dial tcp 10.134.0.12:8443: connect: connection refused"
Aug 04 22:26:18.116565 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.116515    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd" podStartSLOduration=54.174596899 podStartE2EDuration="57.116499616s" totalImagesPullingTime="2.941902717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:26:18.115074293 +0000 UTC m=+315.656567468" watchObservedRunningTime="2026-08-04 22:26:18.116499616 +0000 UTC m=+315.657992792"
Aug 04 22:26:18.689770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.689735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:18.786348 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.786315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:18.838260 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:18.838228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.010646 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.010577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.108790 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.108766    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wzprd"
Aug 04 22:26:19.150125 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.150081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.204866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.204838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.550254 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.550212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.574124 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.574104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:26:19.602526 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.602498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.761647 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.761618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:19.965812 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:19.965778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:20.402379 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:20.402337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:20.559948 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:20.559861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:21.056528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:21.056489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:21.771151 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:21.771109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:21.823963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:21.823933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:22.860312 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:22.860254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:22.960723 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:22.960686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:29.378116 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:29.378073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:29.444372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:29.444349    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:31.785207 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:31.785167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:31.841304 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:31.841214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:34.389685 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:34.389644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:34.445402 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:34.445367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:35.382520 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:35.382489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:37.857303 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:37.857269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:37.967316 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:37.967283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:40.380404 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:40.380370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:40.434554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:40.434523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:41.191616 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:41.191586    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb8717d6-9bab-47ff-be87-16266aa3e79d" containerID="1db73ca072085ecf1f845e242d3633f6fe751730092d285f9a13b58af13258bc" exitCode=0
Aug 04 22:26:41.191787 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:41.191634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fj4tc" event={"ID":"eb8717d6-9bab-47ff-be87-16266aa3e79d","Type":"ContainerDied","Data":"1db73ca072085ecf1f845e242d3633f6fe751730092d285f9a13b58af13258bc"}
Aug 04 22:26:41.192017 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:41.192000    2582 scope.go:117] "RemoveContainer" containerID="1db73ca072085ecf1f845e242d3633f6fe751730092d285f9a13b58af13258bc"
Aug 04 22:26:42.198305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:42.198265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fj4tc" event={"ID":"eb8717d6-9bab-47ff-be87-16266aa3e79d","Type":"ContainerStarted","Data":"83897d71ecfd41a4d2f1d68fdcfbd341c061ff98baa7849d8827c026e5f2daa0"}
Aug 04 22:26:44.587015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:44.586989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-state-metrics/0.log"
Aug 04 22:26:44.786143 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:44.786111    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-rbac-proxy-main/0.log"
Aug 04 22:26:44.912429 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:44.912405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:44.986345 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:44.986320    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-rbac-proxy-self/0.log"
Aug 04 22:26:45.015126 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:45.015088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:45.064765 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:45.064738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:45.386058 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:45.386034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-6rgg2_da3b55d0-6766-4865-afeb-7241dd4a5e88/monitoring-plugin/0.log"
Aug 04 22:26:46.186380 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:46.186354    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/init-textfile/0.log"
Aug 04 22:26:46.387268 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:46.387241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/node-exporter/0.log"
Aug 04 22:26:46.586194 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:46.586114    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/kube-rbac-proxy/0.log"
Aug 04 22:26:47.386323 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:47.386285    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/kube-rbac-proxy-main/0.log"
Aug 04 22:26:47.586340 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:47.586312    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/kube-rbac-proxy-self/0.log"
Aug 04 22:26:47.787435 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:47.787333    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/openshift-state-metrics/0.log"
Aug 04 22:26:47.987186 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:47.987158    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/init-config-reloader/0.log"
Aug 04 22:26:48.188259 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:48.188226    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/prometheus/0.log"
Aug 04 22:26:48.389029 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:48.388999    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/config-reloader/0.log"
Aug 04 22:26:48.587688 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:48.587607    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/thanos-sidecar/0.log"
Aug 04 22:26:48.786417 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:48.786386    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/kube-rbac-proxy-web/0.log"
Aug 04 22:26:48.986329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:48.986302    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/kube-rbac-proxy/0.log"
Aug 04 22:26:49.186458 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:49.186433    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa3ce60-50bd-46e7-b822-0468b2b4e66d/kube-rbac-proxy-thanos/0.log"
Aug 04 22:26:50.587160 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:50.587131    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/thanos-query/0.log"
Aug 04 22:26:50.786188 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:50.786148    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-web/0.log"
Aug 04 22:26:50.986130 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:50.986095    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy/0.log"
Aug 04 22:26:51.187096 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.187068    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/prom-label-proxy/0.log"
Aug 04 22:26:51.236930 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.236853    2582 generic.go:356] "Generic (PLEG): container finished" podID="56f5848b-6584-4442-b731-8d93865c3a11" containerID="7cba3891011e28f918ed0b8acf91360eb5902598b8bbe51784bda5e06980634b" exitCode=0
Aug 04 22:26:51.237042 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.236929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k" event={"ID":"56f5848b-6584-4442-b731-8d93865c3a11","Type":"ContainerDied","Data":"7cba3891011e28f918ed0b8acf91360eb5902598b8bbe51784bda5e06980634b"}
Aug 04 22:26:51.237266 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.237250    2582 scope.go:117] "RemoveContainer" containerID="7cba3891011e28f918ed0b8acf91360eb5902598b8bbe51784bda5e06980634b"
Aug 04 22:26:51.386235 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.386208    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-rules/0.log"
Aug 04 22:26:51.587308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.587226    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-metrics/0.log"
Aug 04 22:26:51.986101 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:51.986074    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:26:52.188999 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:52.188965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/3.log"
Aug 04 22:26:52.242566 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:52.242502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-8jc8k" event={"ID":"56f5848b-6584-4442-b731-8d93865c3a11","Type":"ContainerStarted","Data":"a34c3f10cef82cf5da779dd08bc6ba18af22a264c386dad893a036b80b697aff"}
Aug 04 22:26:52.986809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:52.986719    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67655c4d4d-kzn5x_a520a83c-b775-4e8d-8236-f85692310229/router/0.log"
Aug 04 22:26:53.385694 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:53.385663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqhwj_8fbf8a0c-ad84-445a-beab-7a2f37afd2a7/serve-healthcheck-canary/0.log"
Aug 04 22:26:55.388933 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:55.388866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:26:55.443266 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:26:55.443232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:07.854859 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:07.854820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:07.908794 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:07.908753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:09.574494 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:09.574464    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:09.590575 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:09.590552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:09.887768 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:09.887731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:09.944161 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:09.944126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:10.340718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:10.340645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:12.008849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:12.008812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:12.064358 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:12.064325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:16.813190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:16.813158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:27:16.815559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:16.815537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c5ee73-fd6c-4f21-a662-ec218702b7d9-metrics-certs\") pod \"network-metrics-daemon-bqcgb\" (UID: \"03c5ee73-fd6c-4f21-a662-ec218702b7d9\") " pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:27:16.909569 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:16.909541    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5596r\""
Aug 04 22:27:16.918374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:16.918354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqcgb"
Aug 04 22:27:17.049558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:17.049481    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqcgb"]
Aug 04 22:27:17.051705 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:27:17.051678    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c5ee73_fd6c_4f21_a662_ec218702b7d9.slice/crio-c9aefaa97077b4332664cfd6cf0e771c912c5c80b55a6b0d473924c05ebefe28 WatchSource:0}: Error finding container c9aefaa97077b4332664cfd6cf0e771c912c5c80b55a6b0d473924c05ebefe28: Status 404 returned error can't find the container with id c9aefaa97077b4332664cfd6cf0e771c912c5c80b55a6b0d473924c05ebefe28
Aug 04 22:27:17.352034 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:17.351999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqcgb" event={"ID":"03c5ee73-fd6c-4f21-a662-ec218702b7d9","Type":"ContainerStarted","Data":"c9aefaa97077b4332664cfd6cf0e771c912c5c80b55a6b0d473924c05ebefe28"}
Aug 04 22:27:18.357880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:18.357788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqcgb" event={"ID":"03c5ee73-fd6c-4f21-a662-ec218702b7d9","Type":"ContainerStarted","Data":"ad6f70bb161e5ecd3eb87faa2ea51cf8aebd23f5410d0ba2b602ac884dabb733"}
Aug 04 22:27:18.357880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:18.357851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqcgb" event={"ID":"03c5ee73-fd6c-4f21-a662-ec218702b7d9","Type":"ContainerStarted","Data":"3a6d871f313b22de33f07a297e00f3bf0464efb4eccc9a5ae385206042b9b476"}
Aug 04 22:27:18.373044 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:18.373000    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bqcgb" podStartSLOduration=374.478063016 podStartE2EDuration="6m15.372983534s" totalImagesPullingTime="894.920518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:27:18.372081059 +0000 UTC m=+375.913574229" watchObservedRunningTime="2026-08-04 22:27:18.372983534 +0000 UTC m=+375.914476706"
Aug 04 22:27:18.456105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:18.456066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:18.559496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:18.559461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:19.099110 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:19.099071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:19.152439 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:19.152403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:27.563690 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.563660    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.563966    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="prometheus" containerID="cri-o://e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889" gracePeriod=600
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.564005    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy" containerID="cri-o://f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc" gracePeriod=600
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.564024    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="thanos-sidecar" containerID="cri-o://d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16" gracePeriod=600
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.564040    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1" gracePeriod=600
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.564056    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="config-reloader" containerID="cri-o://397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb" gracePeriod=600
Aug 04 22:27:27.564225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.564083    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-web" containerID="cri-o://849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99" gracePeriod=600
Aug 04 22:27:27.801807 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.801783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:27.907045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907010    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907045 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.907614 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907523    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.908005 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.908005 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.908005 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.908005 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.907778    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.908264 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.908416 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s7vw\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908674    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908706    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908775    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.908913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\" (UID: \"8aa3ce60-50bd-46e7-b822-0468b2b4e66d\") "
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909487    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-metrics-client-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909511    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909525    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909538    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909551    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.909625 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909563    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-k8s-db\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:27.910718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.909730    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.910718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.910074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:27:27.910718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.910144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.911392 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.911327    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.911392 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.911350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.911542 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.911427    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.911628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.911600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config" (OuterVolumeSpecName: "config") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.912022 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.911996    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw" (OuterVolumeSpecName: "kube-api-access-4s7vw") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "kube-api-access-4s7vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:27:27.912228 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.912192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.912291 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.912265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out" (OuterVolumeSpecName: "config-out") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:27:27.912675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.912649    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.913081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.913056    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:27.913349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.913336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:27:27.923131 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:27.923109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config" (OuterVolumeSpecName: "web-config") pod "8aa3ce60-50bd-46e7-b822-0468b2b4e66d" (UID: "8aa3ce60-50bd-46e7-b822-0468b2b4e66d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:27:28.010209 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010180    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7vw\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-kube-api-access-4s7vw\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010209 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010207    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010219    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-tls-assets\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010228    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010236    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-web-config\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010245    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010253    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config-out\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010261    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-kube-rbac-proxy\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010270    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010281    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010289    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-metrics-client-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010297    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-grpc-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010304    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.010394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.010313    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa3ce60-50bd-46e7-b822-0468b2b4e66d-config\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:27:28.401115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401084    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1" exitCode=0
Aug 04 22:27:28.401115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401110    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc" exitCode=0
Aug 04 22:27:28.401115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401118    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99" exitCode=0
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401125    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16" exitCode=0
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401133    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb" exitCode=0
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401141    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889" exitCode=0
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"}
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401237    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"}
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"}
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"}
Aug 04 22:27:28.401329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401322    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.401636 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"}
Aug 04 22:27:28.401636 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"}
Aug 04 22:27:28.401636 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.401384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa3ce60-50bd-46e7-b822-0468b2b4e66d","Type":"ContainerDied","Data":"ae8bf19a01bd58e3c6895e5432d3330a84e20f4d88e9719284e1ddbbb4bf2349"}
Aug 04 22:27:28.412240 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.412217    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.420792 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.420770    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.427672 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.427637    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:28.429076 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.428937    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.431345 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.431320    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:28.436656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.436637    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.443818 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.443797    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.451942 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.451921    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.458800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.458742    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.459180 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.459152    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.459288 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459192    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.459288 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459262    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.459631 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.459608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.459737 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459637    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.459737 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459658    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.459956 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.459934    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.460072 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459960    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.460072 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.459982    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.460430 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.460410    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.460528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.460438    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.460528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.460458    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.460769 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.460733    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.460866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.460770    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.460866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.460791    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.461168 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.461129    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.461168 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461158    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.461401 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461176    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.461401 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461318    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:28.461564 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:27:28.461399    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.461564 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461422    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.461564 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461439    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461633    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461653    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461705    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="prometheus"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461720    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="prometheus"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461733    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerName="registry"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461741    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerName="registry"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461753    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461761    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461772    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="init-config-reloader"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461781    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="init-config-reloader"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461788    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-thanos"
Aug 04 22:27:28.461803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461796    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-thanos"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461825    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="config-reloader"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461833    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="config-reloader"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461828    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461843    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="thanos-sidecar"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461845    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461850    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="thanos-sidecar"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461859    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-web"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461867    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-web"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461949    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="config-reloader"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461965    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="prometheus"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461973    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ec3d1d-5694-4598-9ec9-dd3c18748148" containerName="registry"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461982    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="thanos-sidecar"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461989    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-web"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.461996    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462004    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" containerName="kube-rbac-proxy-thanos"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462052    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462070    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462241    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462260    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462430    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462450    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462610    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.462667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462626    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462787    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462803    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.462996    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463014    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463180    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463196    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463355    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463383    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463547    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463563    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.463759    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.464001    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.464025    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.465696    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.465976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.465719    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.466481 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466021    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.466481 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466039    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.466481 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466257    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.466481 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466273    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.466660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466485    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.466660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466498    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.466747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466711    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.466747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466726    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.466963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466944    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.467014 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.466964    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.467169 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467153    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.467210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467170    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.467425 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467407    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.467474 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467426    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.467667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467648    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.467721 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467677    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.467885 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467865    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.468010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467887    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.468010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.467947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.468101 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468085    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.468154 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468101    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.468669 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468642    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.468732 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468673    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.468973 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468952    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.469020 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.468976    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.469245 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469192    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.469317 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469247    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.469463 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469445    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.469504 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469466    2582 scope.go:117] "RemoveContainer" containerID="4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"
Aug 04 22:27:28.469661 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469645    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1"} err="failed to get container status \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": rpc error: code = NotFound desc = could not find container \"4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1\": container with ID starting with 4cc604f60877b02579efebfdd97218b15082601fe0cd2ef83d8ab873461d28d1 not found: ID does not exist"
Aug 04 22:27:28.469723 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469661    2582 scope.go:117] "RemoveContainer" containerID="f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"
Aug 04 22:27:28.469907 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469873    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc"} err="failed to get container status \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": rpc error: code = NotFound desc = could not find container \"f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc\": container with ID starting with f805fcdbd3fba6edf403acd7e10dece63c28f245c8ff31e56746f5de2fa157bc not found: ID does not exist"
Aug 04 22:27:28.469967 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.469918    2582 scope.go:117] "RemoveContainer" containerID="849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"
Aug 04 22:27:28.470248 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470220    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99"} err="failed to get container status \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": rpc error: code = NotFound desc = could not find container \"849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99\": container with ID starting with 849073a775aa388200957029e028f910c17346ef850cbba53d39afb577fe2d99 not found: ID does not exist"
Aug 04 22:27:28.470309 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470250    2582 scope.go:117] "RemoveContainer" containerID="d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"
Aug 04 22:27:28.470361 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470343    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 22:27:28.470413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470361    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-de1s36g4auta8\""
Aug 04 22:27:28.470413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470402    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 22:27:28.470511 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 22:27:28.470576 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470546    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16"} err="failed to get container status \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": rpc error: code = NotFound desc = could not find container \"d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16\": container with ID starting with d252463a9cfb7cc5fd4ac957e9ea2097faaf2c7fe26d2b9346b3806e57afbd16 not found: ID does not exist"
Aug 04 22:27:28.470628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470579    2582 scope.go:117] "RemoveContainer" containerID="397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"
Aug 04 22:27:28.470719 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470696    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 22:27:28.470800 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 22:27:28.470863 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470833    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb"} err="failed to get container status \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": rpc error: code = NotFound desc = could not find container \"397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb\": container with ID starting with 397e6cab4050c60fa2e9822073ba3f3bbfea3d8b6b86e4017d073751720e39bb not found: ID does not exist"
Aug 04 22:27:28.470863 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470854    2582 scope.go:117] "RemoveContainer" containerID="e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"
Aug 04 22:27:28.470988 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470874    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 22:27:28.471042 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.470998    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tpprg\""
Aug 04 22:27:28.471231 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 22:27:28.471350 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471296    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889"} err="failed to get container status \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": rpc error: code = NotFound desc = could not find container \"e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889\": container with ID starting with e4ffd6b42c2fe94d96672a1c48fdad20effd17506ccdba361ae4d45a174b0889 not found: ID does not exist"
Aug 04 22:27:28.471350 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471319    2582 scope.go:117] "RemoveContainer" containerID="3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"
Aug 04 22:27:28.471498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471382    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471710    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251"} err="failed to get container status \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": rpc error: code = NotFound desc = could not find container \"3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251\": container with ID starting with 3acd2cc362bc02c9c90e7ade203ab10eec6448113ebf192fe7159ecacd2d3251 not found: ID does not exist"
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471718    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471807    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 22:27:28.471843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.471813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 22:27:28.474720 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.474699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 22:27:28.476254 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.476237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 22:27:28.481611 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.481591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:28.615491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8fc\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-kube-api-access-kk8fc\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.615936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.616486 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.615983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.616486 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.616016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.616486 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.616039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.716868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.716868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.716868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.716868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.716868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.716985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8fc\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-kube-api-access-kk8fc\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.717840 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.717656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.718407 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.718384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.720324 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.720010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config-out\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.720324 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.720273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.720616 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.720593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.720616 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.720609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.720741 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.720596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.721705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.721310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.721705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.721514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.721705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.721654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.722144 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.722121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.722238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.722204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.722626 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.722606    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.723100 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.723081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.723347 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.723327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.723423 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.723408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-web-config\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.723809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.723785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.729024 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.728999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8fc\" (UniqueName: \"kubernetes.io/projected/bddc87b2-a2f2-427f-b828-ec2a1c9abfb4-kube-api-access-kk8fc\") pod \"prometheus-k8s-0\" (UID: \"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.778218 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.778184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:28.920046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:28.920005    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 22:27:28.927359 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:27:28.927329    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddc87b2_a2f2_427f_b828_ec2a1c9abfb4.slice/crio-8c85d9b544bb620f453b6146260a71b93a25032ec40fa5997ee2d36c00f7b07b WatchSource:0}: Error finding container 8c85d9b544bb620f453b6146260a71b93a25032ec40fa5997ee2d36c00f7b07b: Status 404 returned error can't find the container with id 8c85d9b544bb620f453b6146260a71b93a25032ec40fa5997ee2d36c00f7b07b
Aug 04 22:27:29.009751 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:29.009721    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa3ce60-50bd-46e7-b822-0468b2b4e66d" path="/var/lib/kubelet/pods/8aa3ce60-50bd-46e7-b822-0468b2b4e66d/volumes"
Aug 04 22:27:29.407457 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:29.407420    2582 generic.go:356] "Generic (PLEG): container finished" podID="bddc87b2-a2f2-427f-b828-ec2a1c9abfb4" containerID="db7ae1fd3c812a4e0dc748fb4b4477a3c89c36bed17027d1bf8777c933eb7e5a" exitCode=0
Aug 04 22:27:29.407601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:29.407509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerDied","Data":"db7ae1fd3c812a4e0dc748fb4b4477a3c89c36bed17027d1bf8777c933eb7e5a"}
Aug 04 22:27:29.407601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:29.407576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"8c85d9b544bb620f453b6146260a71b93a25032ec40fa5997ee2d36c00f7b07b"}
Aug 04 22:27:30.415425 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415387    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"dcc343bf5d281a791e3991b8eaec0bd88757779d5f23f44e02e65a65b38cc8bb"}
Aug 04 22:27:30.415864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"b102793c8e7f624fa3a966604edc87b382bc2bf499c8e3367431bc00379d9f7c"}
Aug 04 22:27:30.415864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"b211142d4f94c6d946d14216f091e853e7e49ed07e8cebbac38f850b3c474400"}
Aug 04 22:27:30.415864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"c93d58c40f64cc52bd9efd85cc8354fb2e9f38864aa4ad477a5e38da860dd171"}
Aug 04 22:27:30.415864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"ac38d82d61fe3cc39d04d8d0b19ea09dfb02de5e615e8b9f6414c9abc1f818d3"}
Aug 04 22:27:30.415864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.415547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bddc87b2-a2f2-427f-b828-ec2a1c9abfb4","Type":"ContainerStarted","Data":"a0c609921e83733bb6534dc23440ccee25015ac82f28aae6d4052e3f11b06528"}
Aug 04 22:27:30.450106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:30.450050    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.428690522 podStartE2EDuration="2.450032221s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.341699ms" isStatefulPod=false podCreationTimestamp="2026-08-04 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:27:30.44919953 +0000 UTC m=+387.990692743" watchObservedRunningTime="2026-08-04 22:27:30.450032221 +0000 UTC m=+387.991525394"
Aug 04 22:27:31.004059 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:31.004024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:31.069915 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:31.069854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:32.043179 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:32.043139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:32.144650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:32.144612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:33.779312 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:33.779276    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:27:37.867506 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:37.867469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:37.967930 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:37.967878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:42.033753 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:42.033723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:42.138853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:42.138819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:43.078979 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:43.078943    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:43.134617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:43.134575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:27:43.185464 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:27:43.185432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:07.142654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:07.142618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:07.238862 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:07.238824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:12.301731 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:12.301678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:28.779346 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:28.779270    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:28:28.795153 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:28.795132    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:28:28.807630 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:28.807608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:29.581498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:29.581439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:29.675492 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:29.675465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 22:28:32.426245 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:32.426204    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:32.526660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:32.526627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:34.891862 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:34.891827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:34.991746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:34.991706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:35.604159 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:35.604115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:35.707305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:35.707278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:36.889447 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:36.889415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:37.290927 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:37.290829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:38.618368 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:38.618338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:38.718330 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:38.718298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:43.570385 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:43.570346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:45.691326 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:45.691289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:46.345669 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.345637    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqtng"]
Aug 04 22:28:46.350161 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.350144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.353083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.353061    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Aug 04 22:28:46.353210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.353069    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Aug 04 22:28:46.353790 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.353774    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5746z\""
Aug 04 22:28:46.359113 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.359093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqtng"]
Aug 04 22:28:46.508219 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.508190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:46.524217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.524193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndhg\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-kube-api-access-fndhg\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.524306 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.524226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.601668 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.601584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:46.625618 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.625594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.625712 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.625662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndhg\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-kube-api-access-fndhg\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.633577 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.633548    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-bound-sa-token\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.633660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.633608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndhg\" (UniqueName: \"kubernetes.io/projected/f05e93fd-096b-4873-990d-c47e596eaedd-kube-api-access-fndhg\") pod \"cert-manager-55b4d954d5-xqtng\" (UID: \"f05e93fd-096b-4873-990d-c47e596eaedd\") " pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.674552 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.674529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xqtng"
Aug 04 22:28:46.804448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:46.804424    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xqtng"]
Aug 04 22:28:46.807333 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:28:46.807304    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf05e93fd_096b_4873_990d_c47e596eaedd.slice/crio-c0ab2e4bff3ff10113a729e726f3e413b41399dd1c4f278cc40427cf70374b93 WatchSource:0}: Error finding container c0ab2e4bff3ff10113a729e726f3e413b41399dd1c4f278cc40427cf70374b93: Status 404 returned error can't find the container with id c0ab2e4bff3ff10113a729e726f3e413b41399dd1c4f278cc40427cf70374b93
Aug 04 22:28:47.738884 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:47.738845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xqtng" event={"ID":"f05e93fd-096b-4873-990d-c47e596eaedd","Type":"ContainerStarted","Data":"c0ab2e4bff3ff10113a729e726f3e413b41399dd1c4f278cc40427cf70374b93"}
Aug 04 22:28:50.722207 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:50.722157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:50.752789 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:50.752762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xqtng" event={"ID":"f05e93fd-096b-4873-990d-c47e596eaedd","Type":"ContainerStarted","Data":"e1ef9ff5f7249788b3093332c64ba66ab297ee47f730431f24016f0735a392e1"}
Aug 04 22:28:50.768754 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:50.768713    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xqtng" podStartSLOduration=0.996317829 podStartE2EDuration="4.768697831s" totalImagesPullingTime="3.772380002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:28:50.767732802 +0000 UTC m=+468.309225975" watchObservedRunningTime="2026-08-04 22:28:50.768697831 +0000 UTC m=+468.310191005"
Aug 04 22:28:50.822109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:50.822081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:52.992173 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:52.992134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:53.130216 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:53.130183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:54.952716 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:54.952679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:28:59.370420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:28:59.370382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:07.060837 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:07.060797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:07.162404 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:07.162371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:07.727292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:07.727258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:07.920857 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:07.920821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.021926 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.021824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.072993 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.072966    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"]
Aug 04 22:29:08.079498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.079478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.081652 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.081632    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Aug 04 22:29:08.081775 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.081669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Aug 04 22:29:08.081775 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.081712    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Aug 04 22:29:08.082043 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.082026    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mqf97\""
Aug 04 22:29:08.082131 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.082079    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Aug 04 22:29:08.091316 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.091297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"]
Aug 04 22:29:08.190745 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.190707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.195776 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.195749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-tmp\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.195864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.195788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.195954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.195865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-manifests\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.196019 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.195972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr776\" (UniqueName: \"kubernetes.io/projected/c9b6ea31-6cb3-443e-98f3-3123dc899aec-kube-api-access-kr776\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.196077 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.196031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296107 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296025    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.296337 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-manifests\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr776\" (UniqueName: \"kubernetes.io/projected/c9b6ea31-6cb3-443e-98f3-3123dc899aec-kube-api-access-kr776\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-tmp\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296522 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.296765 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.296728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-manifests\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.298839 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.298810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b6ea31-6cb3-443e-98f3-3123dc899aec-tmp\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.299015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.298996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.299080 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.299062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9b6ea31-6cb3-443e-98f3-3123dc899aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.308238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.308220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr776\" (UniqueName: \"kubernetes.io/projected/c9b6ea31-6cb3-443e-98f3-3123dc899aec-kube-api-access-kr776\") pod \"opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb\" (UID: \"c9b6ea31-6cb3-443e-98f3-3123dc899aec\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.389956 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.389934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:08.395335 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.395312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.492596 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.492562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.534550 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.534469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"]
Aug 04 22:29:08.537679 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:29:08.537600    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b6ea31_6cb3_443e_98f3_3123dc899aec.slice/crio-105542221f9777b93124787efd64974ea4b5a562c34df72c3376bdafb819dd2c WatchSource:0}: Error finding container 105542221f9777b93124787efd64974ea4b5a562c34df72c3376bdafb819dd2c: Status 404 returned error can't find the container with id 105542221f9777b93124787efd64974ea4b5a562c34df72c3376bdafb819dd2c
Aug 04 22:29:08.595105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.595073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.694958 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.694923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:08.825611 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:08.825537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb" event={"ID":"c9b6ea31-6cb3-443e-98f3-3123dc899aec","Type":"ContainerStarted","Data":"105542221f9777b93124787efd64974ea4b5a562c34df72c3376bdafb819dd2c"}
Aug 04 22:29:11.842100 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:11.842016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb" event={"ID":"c9b6ea31-6cb3-443e-98f3-3123dc899aec","Type":"ContainerStarted","Data":"c458c1331809345eac22c0a6596c0db4ad7804b0be739d17a9022a51b003ebcd"}
Aug 04 22:29:12.853401 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:12.853327    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9b6ea31-6cb3-443e-98f3-3123dc899aec" containerID="c458c1331809345eac22c0a6596c0db4ad7804b0be739d17a9022a51b003ebcd" exitCode=0
Aug 04 22:29:12.853764 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:12.853415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb" event={"ID":"c9b6ea31-6cb3-443e-98f3-3123dc899aec","Type":"ContainerDied","Data":"c458c1331809345eac22c0a6596c0db4ad7804b0be739d17a9022a51b003ebcd"}
Aug 04 22:29:13.859500 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:13.859469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb" event={"ID":"c9b6ea31-6cb3-443e-98f3-3123dc899aec","Type":"ContainerStarted","Data":"0d317b99590300735123e6fe946c466d5382d2ffb5b0b6633aa65cddbac1dd12"}
Aug 04 22:29:13.859916 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:13.859625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:13.884973 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:13.884873    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb" podStartSLOduration=1.468094033 podStartE2EDuration="5.884836881s" totalImagesPullingTime="3.334767539s" totalInitContainerRuntime="1.081975309s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:29:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:29:13.882767441 +0000 UTC m=+491.424260612" watchObservedRunningTime="2026-08-04 22:29:13.884836881 +0000 UTC m=+491.426330056"
Aug 04 22:29:23.449416 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:23.449384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:23.552950 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:23.552854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:24.867479 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:24.867450    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb"
Aug 04 22:29:25.452461 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.452422    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"]
Aug 04 22:29:25.455672 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.455649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.458125 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Aug 04 22:29:25.458216 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458109    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Aug 04 22:29:25.458641 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458623    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Aug 04 22:29:25.458728 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458714    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Aug 04 22:29:25.458772 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-27jgz\""
Aug 04 22:29:25.458772 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.458722    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Aug 04 22:29:25.466175 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.466154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"]
Aug 04 22:29:25.539788 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.539765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-metrics-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.539916 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.539800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-manager-config\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.539979 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.539918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czqs\" (UniqueName: \"kubernetes.io/projected/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-kube-api-access-7czqs\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.539979 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.539946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.640801 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.640774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-metrics-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.640921 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.640806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-manager-config\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.640921 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.640853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czqs\" (UniqueName: \"kubernetes.io/projected/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-kube-api-access-7czqs\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.640921 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.640871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.641537 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.641472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-manager-config\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.643335 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.643306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.643496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.643477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-metrics-cert\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.649201 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.649180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czqs\" (UniqueName: \"kubernetes.io/projected/eefd7f8a-8b34-4484-ba58-54c1dcbff28a-kube-api-access-7czqs\") pod \"lws-controller-manager-76f49b9fdf-rxcsm\" (UID: \"eefd7f8a-8b34-4484-ba58-54c1dcbff28a\") " pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.765398 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.765346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:25.894952 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.894288    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"]
Aug 04 22:29:25.901739 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:29:25.901712    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefd7f8a_8b34_4484_ba58_54c1dcbff28a.slice/crio-d1897e1c0e4771d82d50d4b9a5412610b981daad10480f697db3ba75066f6451 WatchSource:0}: Error finding container d1897e1c0e4771d82d50d4b9a5412610b981daad10480f697db3ba75066f6451: Status 404 returned error can't find the container with id d1897e1c0e4771d82d50d4b9a5412610b981daad10480f697db3ba75066f6451
Aug 04 22:29:25.913571 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:25.913544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm" event={"ID":"eefd7f8a-8b34-4484-ba58-54c1dcbff28a","Type":"ContainerStarted","Data":"d1897e1c0e4771d82d50d4b9a5412610b981daad10480f697db3ba75066f6451"}
Aug 04 22:29:26.975554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:26.975462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:27.129150 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:27.129115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:27.983437 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:27.983379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.043931 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.043879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.145006 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.144963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.245515 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.245426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.549695 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.549599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.648048 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.648013    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.804333 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.804245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:28.931418 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.931388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm" event={"ID":"eefd7f8a-8b34-4484-ba58-54c1dcbff28a","Type":"ContainerStarted","Data":"3f001a4341dfac593e59243a75d753337caf9a568a1ff54491016811110ee740"}
Aug 04 22:29:28.931606 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.931502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:28.948138 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:28.948093    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm" podStartSLOduration=1.19771224 podStartE2EDuration="3.948078589s" totalImagesPullingTime="2.750366349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:29:28.946880138 +0000 UTC m=+506.488373313" watchObservedRunningTime="2026-08-04 22:29:28.948078589 +0000 UTC m=+506.489571763"
Aug 04 22:29:33.008991 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.008951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:33.057956 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.057926    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c576658-pjsgv"]
Aug 04 22:29:33.062096 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.062076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.064588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.064563    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nmvts\""
Aug 04 22:29:33.064716 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.064586    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Aug 04 22:29:33.064847 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.064834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Aug 04 22:29:33.073063 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.073040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c576658-pjsgv"]
Aug 04 22:29:33.211576 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.211534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8d18-834d-4245-b7d0-740a947ff360-tmp\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.211746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.211613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4jd\" (UniqueName: \"kubernetes.io/projected/3fdb8d18-834d-4245-b7d0-740a947ff360-kube-api-access-np4jd\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.211746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.211671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.313147 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.313066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4jd\" (UniqueName: \"kubernetes.io/projected/3fdb8d18-834d-4245-b7d0-740a947ff360-kube-api-access-np4jd\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.313147 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.313126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.313313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.313187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8d18-834d-4245-b7d0-740a947ff360-tmp\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.313313 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:29:33.313286    2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Aug 04 22:29:33.313378 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:29:33.313354    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs podName:3fdb8d18-834d-4245-b7d0-740a947ff360 nodeName:}" failed. No retries permitted until 2026-08-04 22:29:33.813337073 +0000 UTC m=+511.354830229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs") pod "kube-auth-proxy-5c576658-pjsgv" (UID: "3fdb8d18-834d-4245-b7d0-740a947ff360") : secret "kube-auth-proxy-tls" not found
Aug 04 22:29:33.315651 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.315630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fdb8d18-834d-4245-b7d0-740a947ff360-tmp\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.322554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.322528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4jd\" (UniqueName: \"kubernetes.io/projected/3fdb8d18-834d-4245-b7d0-740a947ff360-kube-api-access-np4jd\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.508910 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.508862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:33.817807 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.817778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.819973 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.819955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdb8d18-834d-4245-b7d0-740a947ff360-tls-certs\") pod \"kube-auth-proxy-5c576658-pjsgv\" (UID: \"3fdb8d18-834d-4245-b7d0-740a947ff360\") " pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:33.975592 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:33.975558    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv"
Aug 04 22:29:34.140755 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:34.140729    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c576658-pjsgv"]
Aug 04 22:29:34.143433 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:29:34.143407    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdb8d18_834d_4245_b7d0_740a947ff360.slice/crio-a46b1ac170447425879c57e613a4643e7fd66f71c32ea8aa52a94eb38aad5df5 WatchSource:0}: Error finding container a46b1ac170447425879c57e613a4643e7fd66f71c32ea8aa52a94eb38aad5df5: Status 404 returned error can't find the container with id a46b1ac170447425879c57e613a4643e7fd66f71c32ea8aa52a94eb38aad5df5
Aug 04 22:29:34.279549 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:34.279507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:34.379701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:34.379667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:34.963448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:34.963394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv" event={"ID":"3fdb8d18-834d-4245-b7d0-740a947ff360","Type":"ContainerStarted","Data":"a46b1ac170447425879c57e613a4643e7fd66f71c32ea8aa52a94eb38aad5df5"}
Aug 04 22:29:35.543861 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:35.543079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:36.822415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:36.822339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:37.607208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:37.607172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:37.785309 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:37.785273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:37.844953 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:37.844868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:37.979242 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:37.979213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv" event={"ID":"3fdb8d18-834d-4245-b7d0-740a947ff360","Type":"ContainerStarted","Data":"536c728b7793562f30e730a9464bff393e8691e7e61ff6e87e4d53e4b189e7e3"}
Aug 04 22:29:37.996222 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:37.996171    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c576658-pjsgv" podStartSLOduration=2.180862243 podStartE2EDuration="4.996156433s" totalImagesPullingTime="2.81529419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:29:37.995638343 +0000 UTC m=+515.537131517" watchObservedRunningTime="2026-08-04 22:29:37.996156433 +0000 UTC m=+515.537649607"
Aug 04 22:29:38.065825 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:38.065790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:38.916986 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:38.916945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:39.012712 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:39.012671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:39.126071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:39.126040    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:39.937369 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:39.937342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76f49b9fdf-rxcsm"
Aug 04 22:29:40.355074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:40.354980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:40.567817 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:40.567785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:40.668911 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:40.668854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:29:54.749313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:29:54.749239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:25.975947 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:25.975886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:26.028749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:26.028717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:27.928733 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:27.928689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:27.992492 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:27.992448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:28.700865 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:28.700828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:28.798004 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:28.797971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:41.185699 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:41.185655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:43.772072 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:43.772026    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:44.181866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:44.181830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:44.578629 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:44.578545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:47.005381 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:47.005347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:47.786088 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:47.786051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:48.052804 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:48.052611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:48.179421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:48.179388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:49.701241 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:49.701213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:49.806184 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:49.806159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:50.795342 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:50.795300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:51.021643 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:51.021609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:53.786234 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:53.786192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:53.891009 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:53.890976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:56.482604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:56.482568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:57.620831 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:57.620790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:57.979284 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:57.979207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:58.081281 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:58.081242    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:58.182779 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:58.182744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:30:58.278455 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:30:58.278369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:02.877213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:02.877171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:31:02.877615 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:02.877340    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:31:07.791295 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:07.791257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:07.887464 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:07.887436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:08.471909 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:08.471870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:08.570041 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:08.570000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:12.190602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:12.190558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:12.244504 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:12.244476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:17.574239 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:17.574195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:17.675224 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:17.675171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:18.721288 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:18.721248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:22.191986 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:22.191952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:22.280969 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:22.280939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:22.385665 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:22.385625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:22.484172 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:22.484085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:23.392160 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:23.392085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:23.492604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:23.492567    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:30.615170 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:30.615136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:30.712587 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:30.712531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:33.067730 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:33.067699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:33.120072 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:33.120040    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:37.622272 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:37.622228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:37.721697 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:37.721657    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:38.755871 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:38.755840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:38.813602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:38.813574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:46.844426 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.844390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:46.851845 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.851816    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:31:46.855055 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.855040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:46.857536 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.857514    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 22:31:46.857645 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.857553    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 04 22:31:46.858255 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.858240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tr5x7\""
Aug 04 22:31:46.858314 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.858272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 22:31:46.862747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.862726    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:31:46.949118 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.949090    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:31:46.949295 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.949218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:46.999071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.999035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdxf\" (UniqueName: \"kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:46.999071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:46.999073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.049016 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.048981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:47.100269 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.100184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdxf\" (UniqueName: \"kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.100269 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.100216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.100954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.100936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.109451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.109428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdxf\" (UniqueName: \"kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf\") pod \"limitador-limitador-859c4d96d6-btsxn\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.166344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.166314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:47.305314 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.305266    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:31:47.309906 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:31:47.309865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f59b24f_78a3_4781_81e4_f71c476faad6.slice/crio-1868b515754941d3417ea2024462a9a8a9c4c23897f4caddf354f5af1b6da4f0 WatchSource:0}: Error finding container 1868b515754941d3417ea2024462a9a8a9c4c23897f4caddf354f5af1b6da4f0: Status 404 returned error can't find the container with id 1868b515754941d3417ea2024462a9a8a9c4c23897f4caddf354f5af1b6da4f0
Aug 04 22:31:47.312503 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.312485    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 22:31:47.559705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.559670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" event={"ID":"4f59b24f-78a3-4781-81e4-f71c476faad6","Type":"ContainerStarted","Data":"1868b515754941d3417ea2024462a9a8a9c4c23897f4caddf354f5af1b6da4f0"}
Aug 04 22:31:47.761391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.761353    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:47.837109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.837007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:47.840515 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.840489    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:31:47.844746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.844730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:31:47.847196 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.847173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2f6x4\""
Aug 04 22:31:47.848682 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.848661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:31:47.908502 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:47.908472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vft\" (UniqueName: \"kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft\") pod \"authorino-c548b5d7d-l2j2m\" (UID: \"ac2abdb6-0212-4c34-b511-703b16f2e649\") " pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:31:48.009421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:48.009380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vft\" (UniqueName: \"kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft\") pod \"authorino-c548b5d7d-l2j2m\" (UID: \"ac2abdb6-0212-4c34-b511-703b16f2e649\") " pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:31:48.017836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:48.017803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vft\" (UniqueName: \"kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft\") pod \"authorino-c548b5d7d-l2j2m\" (UID: \"ac2abdb6-0212-4c34-b511-703b16f2e649\") " pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:31:48.155217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:48.155184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:31:48.320412 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:48.320385    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:31:48.323095 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:31:48.322514    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac2abdb6_0212_4c34_b511_703b16f2e649.slice/crio-b5c5365338cc8ac6bdc16e62d8a46c0e784f05eb391044f553092f92a0b666ec WatchSource:0}: Error finding container b5c5365338cc8ac6bdc16e62d8a46c0e784f05eb391044f553092f92a0b666ec: Status 404 returned error can't find the container with id b5c5365338cc8ac6bdc16e62d8a46c0e784f05eb391044f553092f92a0b666ec
Aug 04 22:31:48.567975 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:48.567869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" event={"ID":"ac2abdb6-0212-4c34-b511-703b16f2e649","Type":"ContainerStarted","Data":"b5c5365338cc8ac6bdc16e62d8a46c0e784f05eb391044f553092f92a0b666ec"}
Aug 04 22:31:49.577056 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:49.577008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:49.626799 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:49.626761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:49.693340 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:49.693308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:50.580808 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:50.580772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" event={"ID":"4f59b24f-78a3-4781-81e4-f71c476faad6","Type":"ContainerStarted","Data":"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"}
Aug 04 22:31:50.581206 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:50.580906    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:31:50.599101 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:50.599035    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" podStartSLOduration=2.206037262 podStartE2EDuration="4.599019785s" totalImagesPullingTime="2.392982523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:31:50.597010689 +0000 UTC m=+648.138503857" watchObservedRunningTime="2026-08-04 22:31:50.599019785 +0000 UTC m=+648.140512960"
Aug 04 22:31:52.593844 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:52.593789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" event={"ID":"ac2abdb6-0212-4c34-b511-703b16f2e649","Type":"ContainerStarted","Data":"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"}
Aug 04 22:31:52.608027 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:52.607978    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" podStartSLOduration=1.436381765 podStartE2EDuration="5.607960344s" totalImagesPullingTime="4.171578579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:31:52.60704609 +0000 UTC m=+650.148539288" watchObservedRunningTime="2026-08-04 22:31:52.607960344 +0000 UTC m=+650.149453518"
Aug 04 22:31:54.193388 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:54.193343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:31:54.250714 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:31:54.250664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:00.467019 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:00.466986    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:32:00.467498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:00.467206    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" podUID="4f59b24f-78a3-4781-81e4-f71c476faad6" containerName="limitador" containerID="cri-o://4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2" gracePeriod=30
Aug 04 22:32:00.467836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:00.467818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:32:01.012974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.012952    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:32:01.120094 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.120068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvdxf\" (UniqueName: \"kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf\") pod \"4f59b24f-78a3-4781-81e4-f71c476faad6\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") "
Aug 04 22:32:01.120237 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.120101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file\") pod \"4f59b24f-78a3-4781-81e4-f71c476faad6\" (UID: \"4f59b24f-78a3-4781-81e4-f71c476faad6\") "
Aug 04 22:32:01.120503 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.120478    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file" (OuterVolumeSpecName: "config-file") pod "4f59b24f-78a3-4781-81e4-f71c476faad6" (UID: "4f59b24f-78a3-4781-81e4-f71c476faad6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:32:01.122399 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.122370    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf" (OuterVolumeSpecName: "kube-api-access-qvdxf") pod "4f59b24f-78a3-4781-81e4-f71c476faad6" (UID: "4f59b24f-78a3-4781-81e4-f71c476faad6"). InnerVolumeSpecName "kube-api-access-qvdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:32:01.221048 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.221015    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvdxf\" (UniqueName: \"kubernetes.io/projected/4f59b24f-78a3-4781-81e4-f71c476faad6-kube-api-access-qvdxf\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:01.221048 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.221043    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f59b24f-78a3-4781-81e4-f71c476faad6-config-file\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:01.642101 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.642068    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f59b24f-78a3-4781-81e4-f71c476faad6" containerID="4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2" exitCode=0
Aug 04 22:32:01.642617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.642106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" event={"ID":"4f59b24f-78a3-4781-81e4-f71c476faad6","Type":"ContainerDied","Data":"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"}
Aug 04 22:32:01.642617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.642147    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn"
Aug 04 22:32:01.642617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.642168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-btsxn" event={"ID":"4f59b24f-78a3-4781-81e4-f71c476faad6","Type":"ContainerDied","Data":"1868b515754941d3417ea2024462a9a8a9c4c23897f4caddf354f5af1b6da4f0"}
Aug 04 22:32:01.642617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.642190    2582 scope.go:117] "RemoveContainer" containerID="4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"
Aug 04 22:32:01.651109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.651088    2582 scope.go:117] "RemoveContainer" containerID="4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"
Aug 04 22:32:01.651353 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:01.651334    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2\": container with ID starting with 4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2 not found: ID does not exist" containerID="4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"
Aug 04 22:32:01.651410 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.651361    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2"} err="failed to get container status \"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2\": rpc error: code = NotFound desc = could not find container \"4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2\": container with ID starting with 4e51189af170d76b4f7a5ccaa3609d9ad809dc5b8d87aeea6a1e9fdcb8e783f2 not found: ID does not exist"
Aug 04 22:32:01.662974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.662951    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:32:01.666445 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:01.666426    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-btsxn"]
Aug 04 22:32:02.999405 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:02.999346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:03.010264 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:03.010239    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f59b24f-78a3-4781-81e4-f71c476faad6" path="/var/lib/kubelet/pods/4f59b24f-78a3-4781-81e4-f71c476faad6/volumes"
Aug 04 22:32:03.106879 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:03.106845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:03.489710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:03.489666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:15.274509 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.274464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:15.370656 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.370624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:15.490528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.490499    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gl64j"]
Aug 04 22:32:15.490835 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.490821    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f59b24f-78a3-4781-81e4-f71c476faad6" containerName="limitador"
Aug 04 22:32:15.490880 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.490836    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f59b24f-78a3-4781-81e4-f71c476faad6" containerName="limitador"
Aug 04 22:32:15.490936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.490924    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f59b24f-78a3-4781-81e4-f71c476faad6" containerName="limitador"
Aug 04 22:32:15.495124 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.495109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.500003 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.499980    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gl64j"]
Aug 04 22:32:15.577450 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.577377    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gl64j"]
Aug 04 22:32:15.577650 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:15.577629    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kbkc5], unattached volumes=[], failed to process volumes=[kube-api-access-kbkc5]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-gl64j" podUID="0e685c80-e7c3-435e-bff1-fc92b58b1e31"
Aug 04 22:32:15.601400 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.601370    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-flfb8"]
Aug 04 22:32:15.604859 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.604825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.606916 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.606881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Aug 04 22:32:15.611220 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.611201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-flfb8"]
Aug 04 22:32:15.658255 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.658228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbkc5\" (UniqueName: \"kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5\") pod \"authorino-5dc7ccb5f-gl64j\" (UID: \"0e685c80-e7c3-435e-bff1-fc92b58b1e31\") " pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.715864 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.715836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.722390 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.722368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.738855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.738829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-flfb8"]
Aug 04 22:32:15.739132 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:15.739114    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qvdh2 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-flfb8" podUID="60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e"
Aug 04 22:32:15.756281 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.756254    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:32:15.758985 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.758962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdh2\" (UniqueName: \"kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.759083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.759014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.759137 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.759122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbkc5\" (UniqueName: \"kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5\") pod \"authorino-5dc7ccb5f-gl64j\" (UID: \"0e685c80-e7c3-435e-bff1-fc92b58b1e31\") " pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.759391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.759363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.766849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.766825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:32:15.772077 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.772049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbkc5\" (UniqueName: \"kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5\") pod \"authorino-5dc7ccb5f-gl64j\" (UID: \"0e685c80-e7c3-435e-bff1-fc92b58b1e31\") " pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:15.860093 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.859997    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbkc5\" (UniqueName: \"kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5\") pod \"0e685c80-e7c3-435e-bff1-fc92b58b1e31\" (UID: \"0e685c80-e7c3-435e-bff1-fc92b58b1e31\") "
Aug 04 22:32:15.860283 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.860109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8kz\" (UniqueName: \"kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.860283 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.860226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdh2\" (UniqueName: \"kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.860283 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.860257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.860446 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.860313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.862351 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.862316    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5" (OuterVolumeSpecName: "kube-api-access-kbkc5") pod "0e685c80-e7c3-435e-bff1-fc92b58b1e31" (UID: "0e685c80-e7c3-435e-bff1-fc92b58b1e31"). InnerVolumeSpecName "kube-api-access-kbkc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:32:15.862853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.862830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.867796 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.867777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdh2\" (UniqueName: \"kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2\") pod \"authorino-687bd45585-flfb8\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") " pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:15.961642 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.961601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.961850 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.961691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8kz\" (UniqueName: \"kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.961850 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.961814    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbkc5\" (UniqueName: \"kubernetes.io/projected/0e685c80-e7c3-435e-bff1-fc92b58b1e31-kube-api-access-kbkc5\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:15.964177 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.964155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:15.970305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:15.970283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8kz\" (UniqueName: \"kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz\") pod \"authorino-545b7586dc-xnhf7\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") " pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:16.069708 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.069670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:32:16.200724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.200699    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:32:16.202945 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:32:16.202919    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb75ed4b3_efde_41d4_8251_0b58f3c4c27e.slice/crio-1fc136a5c4a5737a03b9d15c00e99727858400ca45804de2104c56bdcbe2dad8 WatchSource:0}: Error finding container 1fc136a5c4a5737a03b9d15c00e99727858400ca45804de2104c56bdcbe2dad8: Status 404 returned error can't find the container with id 1fc136a5c4a5737a03b9d15c00e99727858400ca45804de2104c56bdcbe2dad8
Aug 04 22:32:16.721885 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.721860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545b7586dc-xnhf7" event={"ID":"b75ed4b3-efde-41d4-8251-0b58f3c4c27e","Type":"ContainerStarted","Data":"1fc136a5c4a5737a03b9d15c00e99727858400ca45804de2104c56bdcbe2dad8"}
Aug 04 22:32:16.721885 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.721880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-gl64j"
Aug 04 22:32:16.722353 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.721913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:16.731966 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.731946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:16.766371 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.766350    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gl64j"]
Aug 04 22:32:16.769875 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.769855    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-gl64j"]
Aug 04 22:32:16.870313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.870278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert\") pod \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") "
Aug 04 22:32:16.870454 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.870345    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvdh2\" (UniqueName: \"kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2\") pod \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\" (UID: \"60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e\") "
Aug 04 22:32:16.872437 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.872410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2" (OuterVolumeSpecName: "kube-api-access-qvdh2") pod "60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e" (UID: "60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e"). InnerVolumeSpecName "kube-api-access-qvdh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:32:16.872549 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.872501    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e" (UID: "60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:32:16.971831 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.971774    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-tls-cert\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:16.971831 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:16.971797    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvdh2\" (UniqueName: \"kubernetes.io/projected/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e-kube-api-access-qvdh2\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:17.010389 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.010360    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e685c80-e7c3-435e-bff1-fc92b58b1e31" path="/var/lib/kubelet/pods/0e685c80-e7c3-435e-bff1-fc92b58b1e31/volumes"
Aug 04 22:32:17.728213 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.728177    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-flfb8"
Aug 04 22:32:17.728600 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.728206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545b7586dc-xnhf7" event={"ID":"b75ed4b3-efde-41d4-8251-0b58f3c4c27e","Type":"ContainerStarted","Data":"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"}
Aug 04 22:32:17.767450 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.767371    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-545b7586dc-xnhf7" podStartSLOduration=2.266585444 podStartE2EDuration="2.767350507s" totalImagesPullingTime="500.765063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:32:17.745436093 +0000 UTC m=+675.286929270" watchObservedRunningTime="2026-08-04 22:32:17.767350507 +0000 UTC m=+675.308843680"
Aug 04 22:32:17.772764 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.772738    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:32:17.772956 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.772927    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" podUID="ac2abdb6-0212-4c34-b511-703b16f2e649" containerName="authorino" containerID="cri-o://a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798" gracePeriod=30
Aug 04 22:32:17.777292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.777261    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-flfb8"]
Aug 04 22:32:17.784011 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:17.783980    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-flfb8"]
Aug 04 22:32:18.026064 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.026038    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:32:18.182836 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.182803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2vft\" (UniqueName: \"kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft\") pod \"ac2abdb6-0212-4c34-b511-703b16f2e649\" (UID: \"ac2abdb6-0212-4c34-b511-703b16f2e649\") "
Aug 04 22:32:18.184885 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.184854    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft" (OuterVolumeSpecName: "kube-api-access-v2vft") pod "ac2abdb6-0212-4c34-b511-703b16f2e649" (UID: "ac2abdb6-0212-4c34-b511-703b16f2e649"). InnerVolumeSpecName "kube-api-access-v2vft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:32:18.283440 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.283366    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vft\" (UniqueName: \"kubernetes.io/projected/ac2abdb6-0212-4c34-b511-703b16f2e649-kube-api-access-v2vft\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:18.733783 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.733751    2582 generic.go:356] "Generic (PLEG): container finished" podID="ac2abdb6-0212-4c34-b511-703b16f2e649" containerID="a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798" exitCode=0
Aug 04 22:32:18.734238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.733793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-l2j2m"
Aug 04 22:32:18.734238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.733827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" event={"ID":"ac2abdb6-0212-4c34-b511-703b16f2e649","Type":"ContainerDied","Data":"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"}
Aug 04 22:32:18.734238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.733870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-l2j2m" event={"ID":"ac2abdb6-0212-4c34-b511-703b16f2e649","Type":"ContainerDied","Data":"b5c5365338cc8ac6bdc16e62d8a46c0e784f05eb391044f553092f92a0b666ec"}
Aug 04 22:32:18.734238 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.733886    2582 scope.go:117] "RemoveContainer" containerID="a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"
Aug 04 22:32:18.742786 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.742770    2582 scope.go:117] "RemoveContainer" containerID="a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"
Aug 04 22:32:18.743055 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:18.743030    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798\": container with ID starting with a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798 not found: ID does not exist" containerID="a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"
Aug 04 22:32:18.743129 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.743059    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798"} err="failed to get container status \"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798\": rpc error: code = NotFound desc = could not find container \"a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798\": container with ID starting with a530a10571be3bc44a86d0daa0f0f0e81f6b9472eaa5d9c847ee64275beee798 not found: ID does not exist"
Aug 04 22:32:18.754518 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.754497    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:32:18.758476 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:18.758459    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-l2j2m"]
Aug 04 22:32:19.009594 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:19.009520    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e" path="/var/lib/kubelet/pods/60f9abf3-f072-4f49-8bc6-fa24b9d8ac8e/volumes"
Aug 04 22:32:19.009745 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:19.009733    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac2abdb6-0212-4c34-b511-703b16f2e649" path="/var/lib/kubelet/pods/ac2abdb6-0212-4c34-b511-703b16f2e649/volumes"
Aug 04 22:32:22.953667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.953634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:22.972208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.972183    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:22.972529 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.972517    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac2abdb6-0212-4c34-b511-703b16f2e649" containerName="authorino"
Aug 04 22:32:22.972578 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.972531    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac2abdb6-0212-4c34-b511-703b16f2e649" containerName="authorino"
Aug 04 22:32:22.972612 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.972590    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac2abdb6-0212-4c34-b511-703b16f2e649" containerName="authorino"
Aug 04 22:32:22.977145 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.977130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:22.980398 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.980365    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Aug 04 22:32:22.980398 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.980360    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Aug 04 22:32:22.980602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.980461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kwzp4\""
Aug 04 22:32:22.986711 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:22.986691    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:23.053574 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.053535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:23.109530 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.109505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:23.125579 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.125552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glbh\" (UniqueName: \"kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.125729 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.125600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.161642 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.161608    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:32:23.165182 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.165160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.175649 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.175625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:32:23.227169 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.227103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glbh\" (UniqueName: \"kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.227169 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.227138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.227370 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:23.227261    2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Aug 04 22:32:23.227370 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:23.227311    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert podName:95c5ef88-1ce8-4d62-85ba-16871faf5c52 nodeName:}" failed. No retries permitted until 2026-08-04 22:32:23.727296211 +0000 UTC m=+681.268789363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert") pod "maas-controller-5d7f7754c7-jr6n6" (UID: "95c5ef88-1ce8-4d62-85ba-16871faf5c52") : secret "maas-controller-webhook-cert" not found
Aug 04 22:32:23.235810 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.235787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glbh\" (UniqueName: \"kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.301910 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.301849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:23.328064 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.328034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqc2\" (UniqueName: \"kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.328169 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.328152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.429387 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.429358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqc2\" (UniqueName: \"kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.429495 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.429444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.431974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.431952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.444029 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.444003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqc2\" (UniqueName: \"kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2\") pod \"maas-controller-7ffd699c46-2677v\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") " pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.478032 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.477980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:23.619067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.619038    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:32:23.620998 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:32:23.620973    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4263a3_0617_48c9_ab3e_ef3593c482e7.slice/crio-7b23de79c77c7459414b2bd98c18b0827586fa9b011724afe51429978ddef830 WatchSource:0}: Error finding container 7b23de79c77c7459414b2bd98c18b0827586fa9b011724afe51429978ddef830: Status 404 returned error can't find the container with id 7b23de79c77c7459414b2bd98c18b0827586fa9b011724afe51429978ddef830
Aug 04 22:32:23.731600 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.731535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.734033 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.734013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") pod \"maas-controller-5d7f7754c7-jr6n6\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") " pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:23.760037 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.760008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd699c46-2677v" event={"ID":"7c4263a3-0617-48c9-ab3e-ef3593c482e7","Type":"ContainerStarted","Data":"7b23de79c77c7459414b2bd98c18b0827586fa9b011724afe51429978ddef830"}
Aug 04 22:32:23.888382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:23.888356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:24.045453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:24.045426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:24.047927 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:32:24.047880    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c5ef88_1ce8_4d62_85ba_16871faf5c52.slice/crio-1a66230a7b624bc5bb5d8b826c9edbfd45f9a7bceaa20f40d4f1615e8dee7743 WatchSource:0}: Error finding container 1a66230a7b624bc5bb5d8b826c9edbfd45f9a7bceaa20f40d4f1615e8dee7743: Status 404 returned error can't find the container with id 1a66230a7b624bc5bb5d8b826c9edbfd45f9a7bceaa20f40d4f1615e8dee7743
Aug 04 22:32:24.770579 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:24.770538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" event={"ID":"95c5ef88-1ce8-4d62-85ba-16871faf5c52","Type":"ContainerStarted","Data":"1a66230a7b624bc5bb5d8b826c9edbfd45f9a7bceaa20f40d4f1615e8dee7743"}
Aug 04 22:32:26.442794 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.442753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:26.541586 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.541492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:26.782170 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.782125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" event={"ID":"95c5ef88-1ce8-4d62-85ba-16871faf5c52","Type":"ContainerStarted","Data":"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"}
Aug 04 22:32:26.782335 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.782207    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:26.783581 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.783556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd699c46-2677v" event={"ID":"7c4263a3-0617-48c9-ab3e-ef3593c482e7","Type":"ContainerStarted","Data":"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"}
Aug 04 22:32:26.783762 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.783741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:26.800698 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.800621    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" podStartSLOduration=2.580756624 podStartE2EDuration="4.800606461s" totalImagesPullingTime="2.219849837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:32:26.799131647 +0000 UTC m=+684.340624822" watchObservedRunningTime="2026-08-04 22:32:26.800606461 +0000 UTC m=+684.342099659"
Aug 04 22:32:26.819070 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:26.819024    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ffd699c46-2677v" podStartSLOduration=1.178599172 podStartE2EDuration="3.819006069s" totalImagesPullingTime="2.640406897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:32:26.818091758 +0000 UTC m=+684.359584931" watchObservedRunningTime="2026-08-04 22:32:26.819006069 +0000 UTC m=+684.360499245"
Aug 04 22:32:37.791720 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:37.791686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:32:37.792212 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:37.791873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:37.848974 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:37.848941    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:37.849133 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:37.849101    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" podUID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" containerName="manager" containerID="cri-o://b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8" gracePeriod=30
Aug 04 22:32:38.097689 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.097667    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:38.146139 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.146111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") pod \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") "
Aug 04 22:32:38.146270 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.146145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2glbh\" (UniqueName: \"kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh\") pod \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\" (UID: \"95c5ef88-1ce8-4d62-85ba-16871faf5c52\") "
Aug 04 22:32:38.148415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.148380    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh" (OuterVolumeSpecName: "kube-api-access-2glbh") pod "95c5ef88-1ce8-4d62-85ba-16871faf5c52" (UID: "95c5ef88-1ce8-4d62-85ba-16871faf5c52"). InnerVolumeSpecName "kube-api-access-2glbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:32:38.148415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.148388    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95c5ef88-1ce8-4d62-85ba-16871faf5c52" (UID: "95c5ef88-1ce8-4d62-85ba-16871faf5c52"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:32:38.247685 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.247655    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95c5ef88-1ce8-4d62-85ba-16871faf5c52-webhook-cert\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:38.247685 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.247681    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glbh\" (UniqueName: \"kubernetes.io/projected/95c5ef88-1ce8-4d62-85ba-16871faf5c52-kube-api-access-2glbh\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:32:38.843228 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.843189    2582 generic.go:356] "Generic (PLEG): container finished" podID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" containerID="b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8" exitCode=0
Aug 04 22:32:38.843623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.843232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" event={"ID":"95c5ef88-1ce8-4d62-85ba-16871faf5c52","Type":"ContainerDied","Data":"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"}
Aug 04 22:32:38.843623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.843246    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6"
Aug 04 22:32:38.843623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.843335    2582 scope.go:117] "RemoveContainer" containerID="b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"
Aug 04 22:32:38.843623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.843324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d7f7754c7-jr6n6" event={"ID":"95c5ef88-1ce8-4d62-85ba-16871faf5c52","Type":"ContainerDied","Data":"1a66230a7b624bc5bb5d8b826c9edbfd45f9a7bceaa20f40d4f1615e8dee7743"}
Aug 04 22:32:38.854585 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.854354    2582 scope.go:117] "RemoveContainer" containerID="b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"
Aug 04 22:32:38.854841 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:38.854814    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8\": container with ID starting with b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8 not found: ID does not exist" containerID="b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"
Aug 04 22:32:38.854960 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.854852    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8"} err="failed to get container status \"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8\": rpc error: code = NotFound desc = could not find container \"b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8\": container with ID starting with b066b5d51652ae5ac65ed91f9722836b8fa1ba65685c9e6ba374cc34a166d8c8 not found: ID does not exist"
Aug 04 22:32:38.863697 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.863673    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:38.867470 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:38.867452    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d7f7754c7-jr6n6"]
Aug 04 22:32:39.010715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:39.010686    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" path="/var/lib/kubelet/pods/95c5ef88-1ce8-4d62-85ba-16871faf5c52/volumes"
Aug 04 22:32:58.053473 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:58.053392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:58.150788 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:58.150746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:59.635757 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.635708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:32:59.696375 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.696347    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"]
Aug 04 22:32:59.696715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.696700    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" containerName="manager"
Aug 04 22:32:59.696757 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.696717    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" containerName="manager"
Aug 04 22:32:59.696795 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.696774    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c5ef88-1ce8-4d62-85ba-16871faf5c52" containerName="manager"
Aug 04 22:32:59.699769 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.699751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.701914 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.701867    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Aug 04 22:32:59.702024 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.701884    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Aug 04 22:32:59.702024 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.701976    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqbsd\""
Aug 04 22:32:59.702719 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.702696    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Aug 04 22:32:59.711420 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.711395    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"]
Aug 04 22:32:59.816524 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.816496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.816694 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.816545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1ee76e-a649-4319-aece-b602bb8795a1-openshift-service-ca\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.816694 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.816624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7fn\" (UniqueName: \"kubernetes.io/projected/0b1ee76e-a649-4319-aece-b602bb8795a1-kube-api-access-hg7fn\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.917999 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.917927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1ee76e-a649-4319-aece-b602bb8795a1-openshift-service-ca\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.917999 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.917989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7fn\" (UniqueName: \"kubernetes.io/projected/0b1ee76e-a649-4319-aece-b602bb8795a1-kube-api-access-hg7fn\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.918162 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.918046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.918162 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:59.918150    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Aug 04 22:32:59.918243 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:32:59.918231    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls podName:0b1ee76e-a649-4319-aece-b602bb8795a1 nodeName:}" failed. No retries permitted until 2026-08-04 22:33:00.418209547 +0000 UTC m=+717.959702700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls") pod "maas-api-6cf454fbc-j2w5m" (UID: "0b1ee76e-a649-4319-aece-b602bb8795a1") : secret "maas-api-serving-cert" not found
Aug 04 22:32:59.918600 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.918581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1ee76e-a649-4319-aece-b602bb8795a1-openshift-service-ca\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:32:59.926525 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:32:59.926499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7fn\" (UniqueName: \"kubernetes.io/projected/0b1ee76e-a649-4319-aece-b602bb8795a1-kube-api-access-hg7fn\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:00.093882 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.093851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:00.191590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.191488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:00.209946 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.209910    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"]
Aug 04 22:33:00.217736 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.217709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.222622 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.222598    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5mbsq\""
Aug 04 22:33:00.222731 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.222619    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"]
Aug 04 22:33:00.223188 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.223171    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Aug 04 22:33:00.250691 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.250668    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x7mv"]
Aug 04 22:33:00.254105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.254086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.266184 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.266162    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x7mv"]
Aug 04 22:33:00.288831 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.288789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:00.322108 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.322079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2828a025-f588-48df-b1eb-a5eb08ea6304-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.322210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.322112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63be74-bd8e-4d9e-8315-0706aeeb283e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.322210 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.322135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqqk\" (UniqueName: \"kubernetes.io/projected/ee63be74-bd8e-4d9e-8315-0706aeeb283e-kube-api-access-2tqqk\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.322283 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.322211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz8qk\" (UniqueName: \"kubernetes.io/projected/2828a025-f588-48df-b1eb-a5eb08ea6304-kube-api-access-mz8qk\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.423190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2828a025-f588-48df-b1eb-a5eb08ea6304-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.423190 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63be74-bd8e-4d9e-8315-0706aeeb283e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.423450 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqqk\" (UniqueName: \"kubernetes.io/projected/ee63be74-bd8e-4d9e-8315-0706aeeb283e-kube-api-access-2tqqk\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.423450 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz8qk\" (UniqueName: \"kubernetes.io/projected/2828a025-f588-48df-b1eb-a5eb08ea6304-kube-api-access-mz8qk\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.423450 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:00.423871 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63be74-bd8e-4d9e-8315-0706aeeb283e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.424012 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.423977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2828a025-f588-48df-b1eb-a5eb08ea6304-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.425969 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.425948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b1ee76e-a649-4319-aece-b602bb8795a1-maas-api-tls\") pod \"maas-api-6cf454fbc-j2w5m\" (UID: \"0b1ee76e-a649-4319-aece-b602bb8795a1\") " pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:00.431285 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.431261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqqk\" (UniqueName: \"kubernetes.io/projected/ee63be74-bd8e-4d9e-8315-0706aeeb283e-kube-api-access-2tqqk\") pod \"payload-processing-5cc6cd444b-5x7mv\" (UID: \"ee63be74-bd8e-4d9e-8315-0706aeeb283e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.431472 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.431452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz8qk\" (UniqueName: \"kubernetes.io/projected/2828a025-f588-48df-b1eb-a5eb08ea6304-kube-api-access-mz8qk\") pod \"payload-pre-processing-687dbb6cf6-6lgcs\" (UID: \"2828a025-f588-48df-b1eb-a5eb08ea6304\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.529655 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.529600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:00.563523 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.563501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:00.612623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.612591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:00.679925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.679672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"]
Aug 04 22:33:00.683611 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:33:00.683571    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2828a025_f588_48df_b1eb_a5eb08ea6304.slice/crio-e3b1e8c2ae61e156ce6d3dcc730d9efe3edb8b4c2cec89ed2b407f7e93d61392 WatchSource:0}: Error finding container e3b1e8c2ae61e156ce6d3dcc730d9efe3edb8b4c2cec89ed2b407f7e93d61392: Status 404 returned error can't find the container with id e3b1e8c2ae61e156ce6d3dcc730d9efe3edb8b4c2cec89ed2b407f7e93d61392
Aug 04 22:33:00.732218 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.732194    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x7mv"]
Aug 04 22:33:00.782195 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.782172    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"]
Aug 04 22:33:00.784598 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:33:00.784569    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1ee76e_a649_4319_aece_b602bb8795a1.slice/crio-4d09e087be1b4584c9363dd6ce2f663e2aa2d91679082b157150695d053a5787 WatchSource:0}: Error finding container 4d09e087be1b4584c9363dd6ce2f663e2aa2d91679082b157150695d053a5787: Status 404 returned error can't find the container with id 4d09e087be1b4584c9363dd6ce2f663e2aa2d91679082b157150695d053a5787
Aug 04 22:33:00.956244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.956206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv" event={"ID":"ee63be74-bd8e-4d9e-8315-0706aeeb283e","Type":"ContainerStarted","Data":"440cd76b2a8e6d52ab75fe1cb4cb2582fc799277d676770f6c5682b7af41a6ad"}
Aug 04 22:33:00.957449 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.957409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m" event={"ID":"0b1ee76e-a649-4319-aece-b602bb8795a1","Type":"ContainerStarted","Data":"4d09e087be1b4584c9363dd6ce2f663e2aa2d91679082b157150695d053a5787"}
Aug 04 22:33:00.958626 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:00.958602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs" event={"ID":"2828a025-f588-48df-b1eb-a5eb08ea6304","Type":"ContainerStarted","Data":"e3b1e8c2ae61e156ce6d3dcc730d9efe3edb8b4c2cec89ed2b407f7e93d61392"}
Aug 04 22:33:03.979132 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.979088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv" event={"ID":"ee63be74-bd8e-4d9e-8315-0706aeeb283e","Type":"ContainerStarted","Data":"b67c50559d546987749685442d968751ef66c1880ba61ce3228ed859ab136499"}
Aug 04 22:33:03.979586 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.979215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:03.980704 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.980679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m" event={"ID":"0b1ee76e-a649-4319-aece-b602bb8795a1","Type":"ContainerStarted","Data":"08b0a9b977c7693421b1ad5e0eb7f35eab63a6a356f236fad145cd724fa4ab7b"}
Aug 04 22:33:03.980843 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.980733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:03.982372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.982340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs" event={"ID":"2828a025-f588-48df-b1eb-a5eb08ea6304","Type":"ContainerStarted","Data":"4a20350ceba414ef74982d0cecefb2505e6b773c0f809600f1eaba1ddee3f4c5"}
Aug 04 22:33:03.982482 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.982399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:03.996301 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:03.996256    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv" podStartSLOduration=1.796878231 podStartE2EDuration="3.996239763s" totalImagesPullingTime="2.199361532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:33:03.993707477 +0000 UTC m=+721.535200658" watchObservedRunningTime="2026-08-04 22:33:03.996239763 +0000 UTC m=+721.537732941"
Aug 04 22:33:04.009620 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:04.009579    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs" podStartSLOduration=1.770145737 podStartE2EDuration="4.009567875s" totalImagesPullingTime="2.239422138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:33:04.007372119 +0000 UTC m=+721.548865295" watchObservedRunningTime="2026-08-04 22:33:04.009567875 +0000 UTC m=+721.551061048"
Aug 04 22:33:04.022227 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:04.022192    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m" podStartSLOduration=2.882829538 podStartE2EDuration="5.02217956s" totalImagesPullingTime="2.139350022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:33:04.021142197 +0000 UTC m=+721.562635373" watchObservedRunningTime="2026-08-04 22:33:04.02217956 +0000 UTC m=+721.563672733"
Aug 04 22:33:09.992143 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:09.992116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf454fbc-j2w5m"
Aug 04 22:33:10.989551 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:10.989497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:11.109453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:11.109419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:14.990083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:14.990052    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-6lgcs"
Aug 04 22:33:14.990527 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:14.990395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5x7mv"
Aug 04 22:33:16.060818 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:16.060783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:16.161466 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:16.161437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:21.881421 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.881372    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"]
Aug 04 22:33:21.886678 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.886655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:21.890217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.889933    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8mcsf\""
Aug 04 22:33:21.890217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.889958    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Aug 04 22:33:21.890217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.889966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Aug 04 22:33:21.890217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.890023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Aug 04 22:33:21.892886 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.892864    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"]
Aug 04 22:33:21.913128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:21.913096    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:22.012972 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.012940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.013142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.012984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf54a-d123-4bad-be45-7c3f704f940b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.013142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.013012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.013142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.013096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j74p\" (UniqueName: \"kubernetes.io/projected/a6ebf54a-d123-4bad-be45-7c3f704f940b-kube-api-access-9j74p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.013142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.013133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.013280 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.013160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.113784 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.113752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.113784 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.113786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf54a-d123-4bad-be45-7c3f704f940b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114068 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.113806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114068 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.113832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j74p\" (UniqueName: \"kubernetes.io/projected/a6ebf54a-d123-4bad-be45-7c3f704f940b-kube-api-access-9j74p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114068 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.113957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114068 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.114003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114289 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.114200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.114294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.114352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.114333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.116296 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.116262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf54a-d123-4bad-be45-7c3f704f940b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.116551 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.116531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf54a-d123-4bad-be45-7c3f704f940b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.120798 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.120780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j74p\" (UniqueName: \"kubernetes.io/projected/a6ebf54a-d123-4bad-be45-7c3f704f940b-kube-api-access-9j74p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd\" (UID: \"a6ebf54a-d123-4bad-be45-7c3f704f940b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.199787 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.199729    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:22.336305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:22.336272    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"]
Aug 04 22:33:22.338258 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:33:22.338221    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ebf54a_d123_4bad_be45_7c3f704f940b.slice/crio-f793f595c5b9707babbfdd8839f820e06aab4c1f8fc4320a788b5f07480eb02d WatchSource:0}: Error finding container f793f595c5b9707babbfdd8839f820e06aab4c1f8fc4320a788b5f07480eb02d: Status 404 returned error can't find the container with id f793f595c5b9707babbfdd8839f820e06aab4c1f8fc4320a788b5f07480eb02d
Aug 04 22:33:23.071109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:23.071063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd" event={"ID":"a6ebf54a-d123-4bad-be45-7c3f704f940b","Type":"ContainerStarted","Data":"f793f595c5b9707babbfdd8839f820e06aab4c1f8fc4320a788b5f07480eb02d"}
Aug 04 22:33:24.077886 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:24.077844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd" event={"ID":"a6ebf54a-d123-4bad-be45-7c3f704f940b","Type":"ContainerStarted","Data":"aea92fce7b1793271f60e5fe2a6a5c760b62105c3eb878094bd29d753d6964d8"}
Aug 04 22:33:24.094134 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:24.094083    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd" podStartSLOduration=1.937566657 podStartE2EDuration="3.094068445s" totalImagesPullingTime="1.156501788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:33:24.09250589 +0000 UTC m=+741.633999077" watchObservedRunningTime="2026-08-04 22:33:24.094068445 +0000 UTC m=+741.635561619"
Aug 04 22:33:32.200011 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:32.199977    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:32.200396 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:32.200050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:32.212817 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:32.212789    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:33.130356 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:33.130328    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd"
Aug 04 22:33:43.229358 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:43.229327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:58.142271 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:58.142232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:58.243944 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:58.243885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:59.093992 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.093956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"]
Aug 04 22:33:59.101003 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.100980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.103369 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.103348    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Aug 04 22:33:59.110224 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.109373    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"]
Aug 04 22:33:59.116514 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.116486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:33:59.220963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.220936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.221338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.220978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df467462-3f0a-4be4-bde3-83baf29a5380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.221338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.221070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.221338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.221106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.221338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.221138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dw8h\" (UniqueName: \"kubernetes.io/projected/df467462-3f0a-4be4-bde3-83baf29a5380-kube-api-access-9dw8h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.221338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.221206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321610 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df467462-3f0a-4be4-bde3-83baf29a5380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dw8h\" (UniqueName: \"kubernetes.io/projected/df467462-3f0a-4be4-bde3-83baf29a5380-kube-api-access-9dw8h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.321770 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.321750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.322057 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.322004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.322126 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.322094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.322225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.322206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.324156 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.324135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df467462-3f0a-4be4-bde3-83baf29a5380-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.324395 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.324378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df467462-3f0a-4be4-bde3-83baf29a5380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.332091 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.332068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dw8h\" (UniqueName: \"kubernetes.io/projected/df467462-3f0a-4be4-bde3-83baf29a5380-kube-api-access-9dw8h\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jhj6n\" (UID: \"df467462-3f0a-4be4-bde3-83baf29a5380\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.414855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.414824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:33:59.547827 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:33:59.547802    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"]
Aug 04 22:33:59.549739 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:33:59.549706    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf467462_3f0a_4be4_bde3_83baf29a5380.slice/crio-66cb8881fdd3450590c6fa0de758d5aa1c604c4c95e7f5e6293a47411d462093 WatchSource:0}: Error finding container 66cb8881fdd3450590c6fa0de758d5aa1c604c4c95e7f5e6293a47411d462093: Status 404 returned error can't find the container with id 66cb8881fdd3450590c6fa0de758d5aa1c604c4c95e7f5e6293a47411d462093
Aug 04 22:34:00.247601 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:00.247564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n" event={"ID":"df467462-3f0a-4be4-bde3-83baf29a5380","Type":"ContainerStarted","Data":"278baf5b256dd856cf58f9955e2b752762a0721473b3cb9836749e54d19f1569"}
Aug 04 22:34:00.248100 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:00.247637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n" event={"ID":"df467462-3f0a-4be4-bde3-83baf29a5380","Type":"ContainerStarted","Data":"66cb8881fdd3450590c6fa0de758d5aa1c604c4c95e7f5e6293a47411d462093"}
Aug 04 22:34:00.269298 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:00.269235    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n" podStartSLOduration=1.084080851 podStartE2EDuration="1.269214943s" totalImagesPullingTime="185.134092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:34:00.268533608 +0000 UTC m=+777.810026808" watchObservedRunningTime="2026-08-04 22:34:00.269214943 +0000 UTC m=+777.810708118"
Aug 04 22:34:09.415746 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:09.415700    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:34:09.416185 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:09.415773    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:34:09.428015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:09.427996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:34:10.304581 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:10.304551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jhj6n"
Aug 04 22:34:31.285300 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:31.285222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:31.528308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:31.528255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:35.925044 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:35.925005    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:36.025813 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:36.025771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:42.591458 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.591421    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"]
Aug 04 22:34:42.596321 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.596296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.598548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.598519    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Aug 04 22:34:42.604795 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.604772    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"]
Aug 04 22:34:42.621532 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.621497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:42.701590 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.701741 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.701741 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.701741 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.701925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edac72f-db15-4507-9758-60f66b0ed7e1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.701925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.701814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzq5\" (UniqueName: \"kubernetes.io/projected/5edac72f-db15-4507-9758-60f66b0ed7e1-kube-api-access-wbzq5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802262 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802262 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edac72f-db15-4507-9758-60f66b0ed7e1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzq5\" (UniqueName: \"kubernetes.io/projected/5edac72f-db15-4507-9758-60f66b0ed7e1-kube-api-access-wbzq5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802666    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.802792 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.802716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.804868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.804845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edac72f-db15-4507-9758-60f66b0ed7e1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.805129 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.805111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edac72f-db15-4507-9758-60f66b0ed7e1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.809640 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.809620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzq5\" (UniqueName: \"kubernetes.io/projected/5edac72f-db15-4507-9758-60f66b0ed7e1-kube-api-access-wbzq5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2f6zw\" (UID: \"5edac72f-db15-4507-9758-60f66b0ed7e1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:42.908803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:42.908776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:43.042877 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:43.042854    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"]
Aug 04 22:34:43.045331 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:34:43.045298    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edac72f_db15_4507_9758_60f66b0ed7e1.slice/crio-47f0e91c610d077d94b8c277e59162238ac75f4c180396641365a92cdb8f4713 WatchSource:0}: Error finding container 47f0e91c610d077d94b8c277e59162238ac75f4c180396641365a92cdb8f4713: Status 404 returned error can't find the container with id 47f0e91c610d077d94b8c277e59162238ac75f4c180396641365a92cdb8f4713
Aug 04 22:34:43.449325 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:43.449275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw" event={"ID":"5edac72f-db15-4507-9758-60f66b0ed7e1","Type":"ContainerStarted","Data":"2c94a53b1eff82b8a6fc0a51911caa919aafbb3ba8c80dbca12ee245f848535e"}
Aug 04 22:34:43.449502 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:43.449343    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw" event={"ID":"5edac72f-db15-4507-9758-60f66b0ed7e1","Type":"ContainerStarted","Data":"47f0e91c610d077d94b8c277e59162238ac75f4c180396641365a92cdb8f4713"}
Aug 04 22:34:43.466543 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:43.466491    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw" podStartSLOduration=1.291139894 podStartE2EDuration="1.466471316s" totalImagesPullingTime="175.331422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:34:43.465020384 +0000 UTC m=+821.006513559" watchObservedRunningTime="2026-08-04 22:34:43.466471316 +0000 UTC m=+821.007964494"
Aug 04 22:34:51.405426 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:51.405371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:34:52.909586 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:52.909543    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:52.910039 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:52.909611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:52.922165 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:52.922140    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:34:53.508126 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:34:53.508099    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2f6zw"
Aug 04 22:35:36.890381 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:36.890348    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78d5c994c8-4gtrk"]
Aug 04 22:35:36.894292 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:36.894269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:36.900628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:36.900603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78d5c994c8-4gtrk"]
Aug 04 22:35:37.073753 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.073725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248b1e83-2492-4ef8-86ab-ac2ecabe2922-tls-cert\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.073915 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.073801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtl6q\" (UniqueName: \"kubernetes.io/projected/248b1e83-2492-4ef8-86ab-ac2ecabe2922-kube-api-access-dtl6q\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.174496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.174433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtl6q\" (UniqueName: \"kubernetes.io/projected/248b1e83-2492-4ef8-86ab-ac2ecabe2922-kube-api-access-dtl6q\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.174496 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.174483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248b1e83-2492-4ef8-86ab-ac2ecabe2922-tls-cert\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.176912 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.176879    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248b1e83-2492-4ef8-86ab-ac2ecabe2922-tls-cert\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.182041 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.182022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtl6q\" (UniqueName: \"kubernetes.io/projected/248b1e83-2492-4ef8-86ab-ac2ecabe2922-kube-api-access-dtl6q\") pod \"authorino-78d5c994c8-4gtrk\" (UID: \"248b1e83-2492-4ef8-86ab-ac2ecabe2922\") " pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.204967 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.204948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78d5c994c8-4gtrk"
Aug 04 22:35:37.336104 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.336020    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78d5c994c8-4gtrk"]
Aug 04 22:35:37.338496 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:35:37.338456    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248b1e83_2492_4ef8_86ab_ac2ecabe2922.slice/crio-9538c348ec4e2be03fad699b116e9e50fabc8e6258952d76344ae457b151267f WatchSource:0}: Error finding container 9538c348ec4e2be03fad699b116e9e50fabc8e6258952d76344ae457b151267f: Status 404 returned error can't find the container with id 9538c348ec4e2be03fad699b116e9e50fabc8e6258952d76344ae457b151267f
Aug 04 22:35:37.709309 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:37.709266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78d5c994c8-4gtrk" event={"ID":"248b1e83-2492-4ef8-86ab-ac2ecabe2922","Type":"ContainerStarted","Data":"9538c348ec4e2be03fad699b116e9e50fabc8e6258952d76344ae457b151267f"}
Aug 04 22:35:38.722010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:38.721887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78d5c994c8-4gtrk" event={"ID":"248b1e83-2492-4ef8-86ab-ac2ecabe2922","Type":"ContainerStarted","Data":"3a39b51c3e5d9fda8ebbc3684f3b2c3b734faa57e0172b4dfc2f5ca6f5499dba"}
Aug 04 22:35:38.738036 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:38.737946    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78d5c994c8-4gtrk" podStartSLOduration=1.6503871270000001 podStartE2EDuration="2.737924458s" totalImagesPullingTime="1.087537331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:35:38.73659995 +0000 UTC m=+876.278093125" watchObservedRunningTime="2026-08-04 22:35:38.737924458 +0000 UTC m=+876.279417633"
Aug 04 22:35:38.768061 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:38.768026    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:35:38.768270 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:38.768241    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-545b7586dc-xnhf7" podUID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" containerName="authorino" containerID="cri-o://663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf" gracePeriod=30
Aug 04 22:35:39.027166 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.027141    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:35:39.192663 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.192632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert\") pod \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") "
Aug 04 22:35:39.192855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.192746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr8kz\" (UniqueName: \"kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz\") pod \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\" (UID: \"b75ed4b3-efde-41d4-8251-0b58f3c4c27e\") "
Aug 04 22:35:39.194970 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.194937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz" (OuterVolumeSpecName: "kube-api-access-qr8kz") pod "b75ed4b3-efde-41d4-8251-0b58f3c4c27e" (UID: "b75ed4b3-efde-41d4-8251-0b58f3c4c27e"). InnerVolumeSpecName "kube-api-access-qr8kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:35:39.207505 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.207468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b75ed4b3-efde-41d4-8251-0b58f3c4c27e" (UID: "b75ed4b3-efde-41d4-8251-0b58f3c4c27e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:35:39.282567 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.282470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:35:39.294259 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.294234    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8kz\" (UniqueName: \"kubernetes.io/projected/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-kube-api-access-qr8kz\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:35:39.294259 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.294258    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b75ed4b3-efde-41d4-8251-0b58f3c4c27e-tls-cert\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:35:39.383349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.383312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:35:39.727774 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.727738    2582 generic.go:356] "Generic (PLEG): container finished" podID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" containerID="663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf" exitCode=0
Aug 04 22:35:39.728246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.727783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545b7586dc-xnhf7"
Aug 04 22:35:39.728246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.727820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545b7586dc-xnhf7" event={"ID":"b75ed4b3-efde-41d4-8251-0b58f3c4c27e","Type":"ContainerDied","Data":"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"}
Aug 04 22:35:39.728246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.727874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545b7586dc-xnhf7" event={"ID":"b75ed4b3-efde-41d4-8251-0b58f3c4c27e","Type":"ContainerDied","Data":"1fc136a5c4a5737a03b9d15c00e99727858400ca45804de2104c56bdcbe2dad8"}
Aug 04 22:35:39.728246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.727953    2582 scope.go:117] "RemoveContainer" containerID="663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"
Aug 04 22:35:39.737568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.737547    2582 scope.go:117] "RemoveContainer" containerID="663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"
Aug 04 22:35:39.737860 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:35:39.737834    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf\": container with ID starting with 663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf not found: ID does not exist" containerID="663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"
Aug 04 22:35:39.737976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.737869    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf"} err="failed to get container status \"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf\": rpc error: code = NotFound desc = could not find container \"663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf\": container with ID starting with 663d4d9e98d46699c105d95f39bebef3fa0dcfb38a9f216ac9bec71ec715f8cf not found: ID does not exist"
Aug 04 22:35:39.748715 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.748692    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:35:39.755413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:39.755372    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-545b7586dc-xnhf7"]
Aug 04 22:35:40.489962 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:40.489925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:35:40.587769 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:40.587735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:35:41.010584 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:41.010554    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" path="/var/lib/kubelet/pods/b75ed4b3-efde-41d4-8251-0b58f3c4c27e/volumes"
Aug 04 22:35:53.968350 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:35:53.968268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:36:02.911003 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:02.910961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:36:02.912745 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:02.912720    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:36:26.552483 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.552443    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:36:26.555108 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.552676    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ffd699c46-2677v" podUID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" containerName="manager" containerID="cri-o://6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c" gracePeriod=30
Aug 04 22:36:26.799113 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.799088    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:36:26.819575 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.819503    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlqc2\" (UniqueName: \"kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2\") pod \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") "
Aug 04 22:36:26.819706 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.819610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert\") pod \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\" (UID: \"7c4263a3-0617-48c9-ab3e-ef3593c482e7\") "
Aug 04 22:36:26.822619 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.822579    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2" (OuterVolumeSpecName: "kube-api-access-jlqc2") pod "7c4263a3-0617-48c9-ab3e-ef3593c482e7" (UID: "7c4263a3-0617-48c9-ab3e-ef3593c482e7"). InnerVolumeSpecName "kube-api-access-jlqc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:36:26.822873 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.822843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7c4263a3-0617-48c9-ab3e-ef3593c482e7" (UID: "7c4263a3-0617-48c9-ab3e-ef3593c482e7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:36:26.921081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.921053    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c4263a3-0617-48c9-ab3e-ef3593c482e7-webhook-cert\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:36:26.921081 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.921078    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqc2\" (UniqueName: \"kubernetes.io/projected/7c4263a3-0617-48c9-ab3e-ef3593c482e7-kube-api-access-jlqc2\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:36:26.953984 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.953954    2582 generic.go:356] "Generic (PLEG): container finished" podID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" containerID="6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c" exitCode=0
Aug 04 22:36:26.954122 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.954016    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd699c46-2677v"
Aug 04 22:36:26.954122 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.954025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd699c46-2677v" event={"ID":"7c4263a3-0617-48c9-ab3e-ef3593c482e7","Type":"ContainerDied","Data":"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"}
Aug 04 22:36:26.954122 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.954067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd699c46-2677v" event={"ID":"7c4263a3-0617-48c9-ab3e-ef3593c482e7","Type":"ContainerDied","Data":"7b23de79c77c7459414b2bd98c18b0827586fa9b011724afe51429978ddef830"}
Aug 04 22:36:26.954122 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.954087    2582 scope.go:117] "RemoveContainer" containerID="6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"
Aug 04 22:36:26.963509 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.963490    2582 scope.go:117] "RemoveContainer" containerID="6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"
Aug 04 22:36:26.963745 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:36:26.963728    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c\": container with ID starting with 6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c not found: ID does not exist" containerID="6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"
Aug 04 22:36:26.963803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.963750    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c"} err="failed to get container status \"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c\": rpc error: code = NotFound desc = could not find container \"6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c\": container with ID starting with 6620673b96a55477bf8161d208fe11f3937e16aeb68d971bed2bbd6591a3e47c not found: ID does not exist"
Aug 04 22:36:26.974489 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.974468    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:36:26.977588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:26.977569    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ffd699c46-2677v"]
Aug 04 22:36:27.010087 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:36:27.010053    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" path="/var/lib/kubelet/pods/7c4263a3-0617-48c9-ab3e-ef3593c482e7/volumes"
Aug 04 22:37:06.408013 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:37:06.407972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:37:38.457655 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:37:38.457624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:37:38.558441 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:37:38.558407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:37:46.960875 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:37:46.960836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:37:48.202207 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:37:48.202171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:47.208920 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:47.208854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:47.308356 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:47.308321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:47.422252 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:47.422213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:53.198349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:53.198268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:53.308547 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:53.308500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:53.408849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:53.408816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:53.532391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:53.532288    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:59.132987 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:59.132948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:38:59.233735 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:38:59.233686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:39:08.941132 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:39:08.941091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:39:08.994192 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:39:08.994154    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:41:02.942713 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:41:02.942638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:41:02.945124 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:41:02.945103    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:43:29.445569 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:43:29.445490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:43:30.570362 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:43:30.570329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:44:17.085510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:44:17.085474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:46:02.974666 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:46:02.974628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:46:02.978373 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:46:02.978344    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:46:54.603936 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:46:54.603847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:48:24.905105 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905074    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:48:24.905559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905443    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" containerName="authorino"
Aug 04 22:48:24.905559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905454    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" containerName="authorino"
Aug 04 22:48:24.905559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905465    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" containerName="manager"
Aug 04 22:48:24.905559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905471    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" containerName="manager"
Aug 04 22:48:24.905559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905556    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75ed4b3-efde-41d4-8251-0b58f3c4c27e" containerName="authorino"
Aug 04 22:48:24.905721 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.905564    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4263a3-0617-48c9-ab3e-ef3593c482e7" containerName="manager"
Aug 04 22:48:24.908614 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.908592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:24.911851 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.911835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8b14625a-kserve-self-signed-certs\""
Aug 04 22:48:24.937623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.937596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:48:24.974130 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:24.974100    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:48:25.024271 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.024397 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.024397 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.024397 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.024547 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5hr\" (UniqueName: \"kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.024547 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.024464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.125675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.125675 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.125803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.125803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5hr\" (UniqueName: \"kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.125803 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.126040 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.125846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.126122 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.126102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.126184 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.126147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.126261 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.126236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.128227 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.128211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.128503 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.128484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.133884 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.133864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5hr\" (UniqueName: \"kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr\") pod \"e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") " pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.217167 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.217100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:25.347391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.347366    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:48:25.349883 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:48:25.349855    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ad169a_164a_45f6_ac9b_9000645c2469.slice/crio-2ca113321cbbd072a78db246580088470ab4a2109bb5654621dbf631a43868cb WatchSource:0}: Error finding container 2ca113321cbbd072a78db246580088470ab4a2109bb5654621dbf631a43868cb: Status 404 returned error can't find the container with id 2ca113321cbbd072a78db246580088470ab4a2109bb5654621dbf631a43868cb
Aug 04 22:48:25.396927 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:25.396876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" event={"ID":"46ad169a-164a-45f6-ac9b-9000645c2469","Type":"ContainerStarted","Data":"2ca113321cbbd072a78db246580088470ab4a2109bb5654621dbf631a43868cb"}
Aug 04 22:48:26.403594 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:26.403559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" event={"ID":"46ad169a-164a-45f6-ac9b-9000645c2469","Type":"ContainerStarted","Data":"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"}
Aug 04 22:48:26.422008 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:26.421964    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" podStartSLOduration=2.421949934 podStartE2EDuration="2.421949934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:48:26.420218685 +0000 UTC m=+1643.961711871" watchObservedRunningTime="2026-08-04 22:48:26.421949934 +0000 UTC m=+1643.963443153"
Aug 04 22:48:34.300734 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.300693    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:48:34.304231 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.304210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.306623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.306603    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8b14625a-kserve-self-signed-certs\""
Aug 04 22:48:34.313660 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.313638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:48:34.334171 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.334145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:48:34.407748 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.407925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.407925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.407925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pbr\" (UniqueName: \"kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.407925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.407925 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.407884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509089 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509243 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509243 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509243 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pbr\" (UniqueName: \"kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509243 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509243 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509577 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.509685 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.509665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.511458 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.511437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.511718 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.511700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.517345 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.517318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pbr\" (UniqueName: \"kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr\") pod \"e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") " pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.614676 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.614620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:34.749249 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:34.749220    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:48:34.751574 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:48:34.751548    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffb2e743_c27c_4888_8bce_f588bff56eb1.slice/crio-8dd852d28c42b8820617518627155e6d9ffb6daaffd3d99013b8c1eb240a5e3a WatchSource:0}: Error finding container 8dd852d28c42b8820617518627155e6d9ffb6daaffd3d99013b8c1eb240a5e3a: Status 404 returned error can't find the container with id 8dd852d28c42b8820617518627155e6d9ffb6daaffd3d99013b8c1eb240a5e3a
Aug 04 22:48:35.217666 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.217593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:35.217666 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.217629    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:35.229808 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.229785    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:35.449829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.449792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" event={"ID":"ffb2e743-c27c-4888-8bce-f588bff56eb1","Type":"ContainerStarted","Data":"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"}
Aug 04 22:48:35.449829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.449831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" event={"ID":"ffb2e743-c27c-4888-8bce-f588bff56eb1","Type":"ContainerStarted","Data":"8dd852d28c42b8820617518627155e6d9ffb6daaffd3d99013b8c1eb240a5e3a"}
Aug 04 22:48:35.461152 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.461126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:48:35.466516 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.466474    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" podStartSLOduration=1.4664596140000001 podStartE2EDuration="1.466459614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:48:35.465758 +0000 UTC m=+1653.007251168" watchObservedRunningTime="2026-08-04 22:48:35.466459614 +0000 UTC m=+1653.007952787"
Aug 04 22:48:35.933271 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:35.933233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:48:44.615318 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:44.615285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:44.615765 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:44.615344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:44.627883 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:44.627860    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:48:45.508846 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:48:45.508818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:49:13.041783 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:13.041749    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:49:13.063299 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:13.063268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:49:14.195727 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:14.195698    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:49:14.215402 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:14.215356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:49:28.061232 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.061145    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" podUID="46ad169a-164a-45f6-ac9b-9000645c2469" containerName="main" containerID="cri-o://6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940" gracePeriod=45
Aug 04 22:49:28.307173 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.307146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:49:28.492474 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5hr\" (UniqueName: \"kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492620    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492863 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492650    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache\") pod \"46ad169a-164a-45f6-ac9b-9000645c2469\" (UID: \"46ad169a-164a-45f6-ac9b-9000645c2469\") "
Aug 04 22:49:28.492863 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home" (OuterVolumeSpecName: "home") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:28.493000 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492960    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache" (OuterVolumeSpecName: "model-cache") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:28.493000 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492977    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.493000 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.492983    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:28.494844 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.494822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm" (OuterVolumeSpecName: "dshm") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:28.494972 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.494887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:49:28.494972 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.494931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr" (OuterVolumeSpecName: "kube-api-access-gb5hr") pod "46ad169a-164a-45f6-ac9b-9000645c2469" (UID: "46ad169a-164a-45f6-ac9b-9000645c2469"). InnerVolumeSpecName "kube-api-access-gb5hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:49:28.593595 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.593569    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.593595 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.593592    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad169a-164a-45f6-ac9b-9000645c2469-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.593760 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.593601    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5hr\" (UniqueName: \"kubernetes.io/projected/46ad169a-164a-45f6-ac9b-9000645c2469-kube-api-access-gb5hr\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.593760 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.593610    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.593760 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.593619    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ad169a-164a-45f6-ac9b-9000645c2469-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:28.706349 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.706313    2582 generic.go:356] "Generic (PLEG): container finished" podID="46ad169a-164a-45f6-ac9b-9000645c2469" containerID="6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940" exitCode=0
Aug 04 22:49:28.706507 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.706364    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"
Aug 04 22:49:28.706507 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.706374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" event={"ID":"46ad169a-164a-45f6-ac9b-9000645c2469","Type":"ContainerDied","Data":"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"}
Aug 04 22:49:28.706507 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.706416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l" event={"ID":"46ad169a-164a-45f6-ac9b-9000645c2469","Type":"ContainerDied","Data":"2ca113321cbbd072a78db246580088470ab4a2109bb5654621dbf631a43868cb"}
Aug 04 22:49:28.706507 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.706440    2582 scope.go:117] "RemoveContainer" containerID="6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"
Aug 04 22:49:28.723341 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.723312    2582 scope.go:117] "RemoveContainer" containerID="6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"
Aug 04 22:49:28.723642 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:49:28.723615    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940\": container with ID starting with 6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940 not found: ID does not exist" containerID="6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"
Aug 04 22:49:28.723698 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.723655    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940"} err="failed to get container status \"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940\": rpc error: code = NotFound desc = could not find container \"6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940\": container with ID starting with 6423ea347dda0fef4c91618c70f022ac76e0bff4e180bb3b26a5d6f402728940 not found: ID does not exist"
Aug 04 22:49:28.735225 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.735200    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:49:28.738144 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:28.738120    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8b14625a-kserve-554f59cc99-k8z2l"]
Aug 04 22:49:29.011148 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.011115    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ad169a-164a-45f6-ac9b-9000645c2469" path="/var/lib/kubelet/pods/46ad169a-164a-45f6-ac9b-9000645c2469/volumes"
Aug 04 22:49:29.215427 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.215368    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" podUID="ffb2e743-c27c-4888-8bce-f588bff56eb1" containerName="main" containerID="cri-o://f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792" gracePeriod=45
Aug 04 22:49:29.473308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.473283    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:49:29.603080 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603021    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603080 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603077    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4pbr\" (UniqueName: \"kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603175    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603410 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home\") pod \"ffb2e743-c27c-4888-8bce-f588bff56eb1\" (UID: \"ffb2e743-c27c-4888-8bce-f588bff56eb1\") "
Aug 04 22:49:29.603410 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:29.603510 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home" (OuterVolumeSpecName: "home") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:29.603564 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603512    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:29.603646 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603629    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.603700 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603652    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.603700 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.603668    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.605244 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.605220    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:49:29.605344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.605250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm" (OuterVolumeSpecName: "dshm") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 22:49:29.605388 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.605340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr" (OuterVolumeSpecName: "kube-api-access-z4pbr") pod "ffb2e743-c27c-4888-8bce-f588bff56eb1" (UID: "ffb2e743-c27c-4888-8bce-f588bff56eb1"). InnerVolumeSpecName "kube-api-access-z4pbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:49:29.705084 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.705040    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pbr\" (UniqueName: \"kubernetes.io/projected/ffb2e743-c27c-4888-8bce-f588bff56eb1-kube-api-access-z4pbr\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.705276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.705257    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb2e743-c27c-4888-8bce-f588bff56eb1-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.705361 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.705284    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb2e743-c27c-4888-8bce-f588bff56eb1-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:49:29.712004 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.711975    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffb2e743-c27c-4888-8bce-f588bff56eb1" containerID="f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792" exitCode=0
Aug 04 22:49:29.712107 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.712018    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"
Aug 04 22:49:29.712107 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.712068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" event={"ID":"ffb2e743-c27c-4888-8bce-f588bff56eb1","Type":"ContainerDied","Data":"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"}
Aug 04 22:49:29.712239 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.712118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz" event={"ID":"ffb2e743-c27c-4888-8bce-f588bff56eb1","Type":"ContainerDied","Data":"8dd852d28c42b8820617518627155e6d9ffb6daaffd3d99013b8c1eb240a5e3a"}
Aug 04 22:49:29.712239 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.712143    2582 scope.go:117] "RemoveContainer" containerID="f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"
Aug 04 22:49:29.722991 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.722975    2582 scope.go:117] "RemoveContainer" containerID="f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"
Aug 04 22:49:29.723239 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:49:29.723208    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792\": container with ID starting with f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792 not found: ID does not exist" containerID="f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"
Aug 04 22:49:29.723342 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.723243    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792"} err="failed to get container status \"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792\": rpc error: code = NotFound desc = could not find container \"f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792\": container with ID starting with f89c84ae9b50a05ff21f398405aa80d689b89c768c6b853096e9db54f4698792 not found: ID does not exist"
Aug 04 22:49:29.733269 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.733249    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:49:29.736372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:29.736351    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8b14625a-kserve-5d5d8b9999-g4pdz"]
Aug 04 22:49:31.009954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:49:31.009923    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb2e743-c27c-4888-8bce-f588bff56eb1" path="/var/lib/kubelet/pods/ffb2e743-c27c-4888-8bce-f588bff56eb1/volumes"
Aug 04 22:51:03.006299 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:51:03.006265    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:51:03.011473 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:51:03.011438    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:52:33.426491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:52:33.426396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:52:59.358154 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:52:59.358105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:02.351866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:02.351812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:02.455419 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:02.455372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:02.822486 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:02.822450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:08.468912 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:08.468865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:08.569109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:08.569069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:08.708555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:08.708520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:09.038535 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:09.038500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:09.373848 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:09.373813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:11.332830 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:11.332793    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:11.432686 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:11.432650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:12.900255 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.900208    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:12.901064 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb2e743-c27c-4888-8bce-f588bff56eb1" containerName="main"
Aug 04 22:53:12.901064 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901066    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb2e743-c27c-4888-8bce-f588bff56eb1" containerName="main"
Aug 04 22:53:12.901247 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901105    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ad169a-164a-45f6-ac9b-9000645c2469" containerName="main"
Aug 04 22:53:12.901247 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ad169a-164a-45f6-ac9b-9000645c2469" containerName="main"
Aug 04 22:53:12.901399 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901373    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb2e743-c27c-4888-8bce-f588bff56eb1" containerName="main"
Aug 04 22:53:12.901613 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.901595    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ad169a-164a-45f6-ac9b-9000645c2469" containerName="main"
Aug 04 22:53:12.905527 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.905505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:12.908546 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.908453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e4f3cae8-serving-cert\""
Aug 04 22:53:12.908976 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.908934    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:12.932554 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:12.932528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:13.058657 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.058623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.058798 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.058666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.058798 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.058701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9kx\" (UniqueName: \"kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.119195 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.119173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:13.159286 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.159219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.159286 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.159264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.159454 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.159310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9kx\" (UniqueName: \"kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.159826 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.159806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.161763 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.161741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.167569 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.167542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9kx\" (UniqueName: \"kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx\") pod \"maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.217676 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.217641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:13.349950 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.349926    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:13.352399 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:53:13.352368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30b2dd6_a380_4dc6_9e11_30124de693bf.slice/crio-60fe5479cfc718923a9fd18b9f96aecbc7b616120e602cebe5b9f2c5e23a8cec WatchSource:0}: Error finding container 60fe5479cfc718923a9fd18b9f96aecbc7b616120e602cebe5b9f2c5e23a8cec: Status 404 returned error can't find the container with id 60fe5479cfc718923a9fd18b9f96aecbc7b616120e602cebe5b9f2c5e23a8cec
Aug 04 22:53:13.353780 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.353762    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 22:53:13.458362 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.458267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:13.505185 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.505153    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:13.509652 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.509633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.512117 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.512097    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e4f3cae8-dockercfg-vkqfl\""
Aug 04 22:53:13.512315 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.512300    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e4f3cae8\""
Aug 04 22:53:13.518558 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.518536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:13.539491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.539454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:13.540002 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.539979    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:13.543368 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.543340    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.558394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.558363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:13.663977 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.663942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.664157 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.664002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x25\" (UniqueName: \"kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.664157 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.664130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.664276 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.664185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2tp\" (UniqueName: \"kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.764964 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.764857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x25\" (UniqueName: \"kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.765115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.764976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.765115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.765016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2tp\" (UniqueName: \"kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.765115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.765072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.765669 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.765647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.765747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.765720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.777623 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.777597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x25\" (UniqueName: \"kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25\") pod \"payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") " pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.777720 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.777623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2tp\" (UniqueName: \"kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp\") pod \"payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.783242 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.783214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" event={"ID":"a30b2dd6-a380-4dc6-9e11-30124de693bf","Type":"ContainerStarted","Data":"60fe5479cfc718923a9fd18b9f96aecbc7b616120e602cebe5b9f2c5e23a8cec"}
Aug 04 22:53:13.821749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.821721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:13.853124 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.853098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:13.967613 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:13.967574    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:13.971610 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:53:13.971499    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625cf266_954f_4d7b_83ea_6fa436183069.slice/crio-23879be7231511066e9e4dd97131d52004a34251aa572409fd576e9329fd9736 WatchSource:0}: Error finding container 23879be7231511066e9e4dd97131d52004a34251aa572409fd576e9329fd9736: Status 404 returned error can't find the container with id 23879be7231511066e9e4dd97131d52004a34251aa572409fd576e9329fd9736
Aug 04 22:53:14.002175 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.002148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:14.004445 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:53:14.004416    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6bfd68_488a_4771_8ec0_9b74d7d0b73e.slice/crio-d360a709a710a55a9ecf072ea7619e8f491f913cce670f6cbcadfa9900227125 WatchSource:0}: Error finding container d360a709a710a55a9ecf072ea7619e8f491f913cce670f6cbcadfa9900227125: Status 404 returned error can't find the container with id d360a709a710a55a9ecf072ea7619e8f491f913cce670f6cbcadfa9900227125
Aug 04 22:53:14.789237 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.789195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" event={"ID":"a30b2dd6-a380-4dc6-9e11-30124de693bf","Type":"ContainerStarted","Data":"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"}
Aug 04 22:53:14.789448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.789365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:14.790732 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.790707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" event={"ID":"625cf266-954f-4d7b-83ea-6fa436183069","Type":"ContainerStarted","Data":"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"}
Aug 04 22:53:14.790849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.790744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" event={"ID":"625cf266-954f-4d7b-83ea-6fa436183069","Type":"ContainerStarted","Data":"23879be7231511066e9e4dd97131d52004a34251aa572409fd576e9329fd9736"}
Aug 04 22:53:14.790849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.790762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:14.792332 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.792305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" event={"ID":"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e","Type":"ContainerStarted","Data":"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"}
Aug 04 22:53:14.792459 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.792345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" event={"ID":"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e","Type":"ContainerStarted","Data":"d360a709a710a55a9ecf072ea7619e8f491f913cce670f6cbcadfa9900227125"}
Aug 04 22:53:14.792459 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.792365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:14.806887 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.806848    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" podStartSLOduration=2.004744013 podStartE2EDuration="2.8068362s" totalImagesPullingTime="802.092187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:53:14.806197358 +0000 UTC m=+1932.347690531" watchObservedRunningTime="2026-08-04 22:53:14.8068362 +0000 UTC m=+1932.348329373"
Aug 04 22:53:14.821374 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.821336    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" podStartSLOduration=1.82132508 podStartE2EDuration="1.82132508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:53:14.820079134 +0000 UTC m=+1932.361572309" watchObservedRunningTime="2026-08-04 22:53:14.82132508 +0000 UTC m=+1932.362818283"
Aug 04 22:53:14.835980 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:14.835948    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" podStartSLOduration=1.835936617 podStartE2EDuration="1.835936617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:53:14.834752395 +0000 UTC m=+1932.376245570" watchObservedRunningTime="2026-08-04 22:53:14.835936617 +0000 UTC m=+1932.377429790"
Aug 04 22:53:20.803433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:20.803401    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:25.799945 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:25.799883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:25.800382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:25.800097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:43.014208 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.014168    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:43.016787 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.014390    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" podUID="a30b2dd6-a380-4dc6-9e11-30124de693bf" containerName="maas-api" containerID="cri-o://0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999" gracePeriod=30
Aug 04 22:53:43.038738 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.038714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:43.063453 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.063423    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:43.063805 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.063749    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" podUID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" containerName="payload-processing" containerID="cri-o://f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383" gracePeriod=30
Aug 04 22:53:43.071602 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.071574    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:43.071801 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.071777    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" podUID="625cf266-954f-4d7b-83ea-6fa436183069" containerName="payload-pre-processing" containerID="cri-o://024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f" gracePeriod=30
Aug 04 22:53:43.386874 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.386849    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:43.390071 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.390050    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:43.400108 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.400089    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:43.416415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416385    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume\") pod \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") "
Aug 04 22:53:43.416415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g2tp\" (UniqueName: \"kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp\") pod \"625cf266-954f-4d7b-83ea-6fa436183069\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") "
Aug 04 22:53:43.416617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume\") pod \"625cf266-954f-4d7b-83ea-6fa436183069\" (UID: \"625cf266-954f-4d7b-83ea-6fa436183069\") "
Aug 04 22:53:43.416617 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89x25\" (UniqueName: \"kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25\") pod \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\" (UID: \"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e\") "
Aug 04 22:53:43.416795 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" (UID: "6a6bfd68-488a-4771-8ec0-9b74d7d0b73e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:53:43.416909 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.416869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "625cf266-954f-4d7b-83ea-6fa436183069" (UID: "625cf266-954f-4d7b-83ea-6fa436183069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:53:43.419631 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.419528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp" (OuterVolumeSpecName: "kube-api-access-9g2tp") pod "625cf266-954f-4d7b-83ea-6fa436183069" (UID: "625cf266-954f-4d7b-83ea-6fa436183069"). InnerVolumeSpecName "kube-api-access-9g2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:53:43.420253 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.420233    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25" (OuterVolumeSpecName: "kube-api-access-89x25") pod "6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" (UID: "6a6bfd68-488a-4771-8ec0-9b74d7d0b73e"). InnerVolumeSpecName "kube-api-access-89x25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:53:43.517919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.517859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls\") pod \"a30b2dd6-a380-4dc6-9e11-30124de693bf\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") "
Aug 04 22:53:43.518106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.517929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q9kx\" (UniqueName: \"kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx\") pod \"a30b2dd6-a380-4dc6-9e11-30124de693bf\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") "
Aug 04 22:53:43.518106 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.517984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca\") pod \"a30b2dd6-a380-4dc6-9e11-30124de693bf\" (UID: \"a30b2dd6-a380-4dc6-9e11-30124de693bf\") "
Aug 04 22:53:43.518310 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.518292    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625cf266-954f-4d7b-83ea-6fa436183069-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.518372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.518315    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x25\" (UniqueName: \"kubernetes.io/projected/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-kube-api-access-89x25\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.518372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.518321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a30b2dd6-a380-4dc6-9e11-30124de693bf" (UID: "a30b2dd6-a380-4dc6-9e11-30124de693bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:53:43.518372 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.518330    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.518474 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.518371    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2tp\" (UniqueName: \"kubernetes.io/projected/625cf266-954f-4d7b-83ea-6fa436183069-kube-api-access-9g2tp\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.520217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.520195    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a30b2dd6-a380-4dc6-9e11-30124de693bf" (UID: "a30b2dd6-a380-4dc6-9e11-30124de693bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:53:43.520263 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.520245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx" (OuterVolumeSpecName: "kube-api-access-9q9kx") pod "a30b2dd6-a380-4dc6-9e11-30124de693bf" (UID: "a30b2dd6-a380-4dc6-9e11-30124de693bf"). InnerVolumeSpecName "kube-api-access-9q9kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:53:43.619495 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.619469    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a30b2dd6-a380-4dc6-9e11-30124de693bf-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.619495 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.619494    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9kx\" (UniqueName: \"kubernetes.io/projected/a30b2dd6-a380-4dc6-9e11-30124de693bf-kube-api-access-9q9kx\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.619654 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.619508    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30b2dd6-a380-4dc6-9e11-30124de693bf-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:53:43.934089 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.934005    2582 generic.go:356] "Generic (PLEG): container finished" podID="625cf266-954f-4d7b-83ea-6fa436183069" containerID="024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f" exitCode=0
Aug 04 22:53:43.934089 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.934052    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"
Aug 04 22:53:43.934327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.934103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" event={"ID":"625cf266-954f-4d7b-83ea-6fa436183069","Type":"ContainerDied","Data":"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"}
Aug 04 22:53:43.934327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.934163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6" event={"ID":"625cf266-954f-4d7b-83ea-6fa436183069","Type":"ContainerDied","Data":"23879be7231511066e9e4dd97131d52004a34251aa572409fd576e9329fd9736"}
Aug 04 22:53:43.934327 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.934193    2582 scope.go:117] "RemoveContainer" containerID="024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"
Aug 04 22:53:43.935438 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.935412    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" containerID="f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383" exitCode=0
Aug 04 22:53:43.935555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.935456    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"
Aug 04 22:53:43.935555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.935503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" event={"ID":"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e","Type":"ContainerDied","Data":"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"}
Aug 04 22:53:43.935665 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.935569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd" event={"ID":"6a6bfd68-488a-4771-8ec0-9b74d7d0b73e","Type":"ContainerDied","Data":"d360a709a710a55a9ecf072ea7619e8f491f913cce670f6cbcadfa9900227125"}
Aug 04 22:53:43.937337 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.937311    2582 generic.go:356] "Generic (PLEG): container finished" podID="a30b2dd6-a380-4dc6-9e11-30124de693bf" containerID="0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999" exitCode=0
Aug 04 22:53:43.937485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.937355    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"
Aug 04 22:53:43.937485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.937379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" event={"ID":"a30b2dd6-a380-4dc6-9e11-30124de693bf","Type":"ContainerDied","Data":"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"}
Aug 04 22:53:43.937485 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.937412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth" event={"ID":"a30b2dd6-a380-4dc6-9e11-30124de693bf","Type":"ContainerDied","Data":"60fe5479cfc718923a9fd18b9f96aecbc7b616120e602cebe5b9f2c5e23a8cec"}
Aug 04 22:53:43.944035 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.944015    2582 scope.go:117] "RemoveContainer" containerID="024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"
Aug 04 22:53:43.944299 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:53:43.944277    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f\": container with ID starting with 024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f not found: ID does not exist" containerID="024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"
Aug 04 22:53:43.944398 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.944308    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f"} err="failed to get container status \"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f\": rpc error: code = NotFound desc = could not find container \"024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f\": container with ID starting with 024c1fd1d575fce8a5a3d71cc9fbda0919f3c4da83fc6174a05fd95b37eac18f not found: ID does not exist"
Aug 04 22:53:43.944398 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.944331    2582 scope.go:117] "RemoveContainer" containerID="f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"
Aug 04 22:53:43.952855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.952839    2582 scope.go:117] "RemoveContainer" containerID="f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"
Aug 04 22:53:43.953132 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:53:43.953114    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383\": container with ID starting with f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383 not found: ID does not exist" containerID="f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"
Aug 04 22:53:43.953187 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.953139    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383"} err="failed to get container status \"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383\": rpc error: code = NotFound desc = could not find container \"f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383\": container with ID starting with f397025c89c484d7aba4d8da7d005a5bd557c36da3ce5f64d17469fd9aa33383 not found: ID does not exist"
Aug 04 22:53:43.953187 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.953155    2582 scope.go:117] "RemoveContainer" containerID="0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"
Aug 04 22:53:43.960522 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.960505    2582 scope.go:117] "RemoveContainer" containerID="0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"
Aug 04 22:53:43.960771 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:53:43.960750    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999\": container with ID starting with 0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999 not found: ID does not exist" containerID="0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"
Aug 04 22:53:43.960860 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.960773    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999"} err="failed to get container status \"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999\": rpc error: code = NotFound desc = could not find container \"0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999\": container with ID starting with 0acebc5f8992db597f5906894a91764aa464e4508fbb88ca5e9e9ddd7b1ab999 not found: ID does not exist"
Aug 04 22:53:43.970565 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.970540    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:43.979777 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.979757    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4f3cae8-db4784564-dwpk6"]
Aug 04 22:53:43.990369 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.990350    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:43.999778 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:43.999760    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4f3cae8-65f6667d58-mlvfd"]
Aug 04 22:53:44.019382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:44.019348    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:44.026551 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:44.026531    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4f3cae8-76b74d8cf5-52qth"]
Aug 04 22:53:44.570985 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:44.570948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:44.793637 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:44.793606    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:45.010433 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:45.010401    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625cf266-954f-4d7b-83ea-6fa436183069" path="/var/lib/kubelet/pods/625cf266-954f-4d7b-83ea-6fa436183069/volumes"
Aug 04 22:53:45.010786 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:45.010773    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" path="/var/lib/kubelet/pods/6a6bfd68-488a-4771-8ec0-9b74d7d0b73e/volumes"
Aug 04 22:53:45.011185 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:45.011169    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30b2dd6-a380-4dc6-9e11-30124de693bf" path="/var/lib/kubelet/pods/a30b2dd6-a380-4dc6-9e11-30124de693bf/volumes"
Aug 04 22:53:46.531627 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.531589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:46.644548 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:53:46.644907 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644881    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" containerName="payload-processing"
Aug 04 22:53:46.644954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644910    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" containerName="payload-processing"
Aug 04 22:53:46.644954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644934    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30b2dd6-a380-4dc6-9e11-30124de693bf" containerName="maas-api"
Aug 04 22:53:46.644954 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644941    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30b2dd6-a380-4dc6-9e11-30124de693bf" containerName="maas-api"
Aug 04 22:53:46.645052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644959    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625cf266-954f-4d7b-83ea-6fa436183069" containerName="payload-pre-processing"
Aug 04 22:53:46.645052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.644968    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="625cf266-954f-4d7b-83ea-6fa436183069" containerName="payload-pre-processing"
Aug 04 22:53:46.645111 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.645050    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30b2dd6-a380-4dc6-9e11-30124de693bf" containerName="maas-api"
Aug 04 22:53:46.645111 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.645066    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="625cf266-954f-4d7b-83ea-6fa436183069" containerName="payload-pre-processing"
Aug 04 22:53:46.645111 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.645078    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6bfd68-488a-4771-8ec0-9b74d7d0b73e" containerName="payload-processing"
Aug 04 22:53:46.649683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.649663    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.659289 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.659272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-63f65db9-serving-cert\""
Aug 04 22:53:46.667916 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.667869    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:53:46.694963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.694920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:46.746817 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.746794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.746931 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.746840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv42\" (UniqueName: \"kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.746931 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.746873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.798427 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.798370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:46.847707 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.847681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv42\" (UniqueName: \"kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.847809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.847722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.847809 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.847779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.848403 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.848383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.850343 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.850320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.860743 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.860715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv42\" (UniqueName: \"kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42\") pod \"maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:46.959973 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:46.959947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:47.131856 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.131829    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:53:47.134004 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:53:47.133979    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f16fad7_b0b6_41a0_9628_5d1b2a1964ed.slice/crio-42744c4a8858eda48ae8f31f6cdead6479af6a40b2ed009ceed561a7d447ad41 WatchSource:0}: Error finding container 42744c4a8858eda48ae8f31f6cdead6479af6a40b2ed009ceed561a7d447ad41: Status 404 returned error can't find the container with id 42744c4a8858eda48ae8f31f6cdead6479af6a40b2ed009ceed561a7d447ad41
Aug 04 22:53:47.558203 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.558175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:47.673146 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.673101    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:53:47.676783 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.676763    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.680046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.680027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-63f65db9-dockercfg-s76zv\""
Aug 04 22:53:47.680397 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.680382    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-63f65db9\""
Aug 04 22:53:47.693157 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.693136    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:53:47.697260 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.697233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:47.755546 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.755516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.755667 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.755559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9mw\" (UniqueName: \"kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.856528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.856452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.856528 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.856510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9mw\" (UniqueName: \"kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.857338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.857296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.866724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.866695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9mw\" (UniqueName: \"kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw\") pod \"payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") " pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.960969 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.960935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" event={"ID":"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed","Type":"ContainerStarted","Data":"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"}
Aug 04 22:53:47.961129 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.960995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" event={"ID":"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed","Type":"ContainerStarted","Data":"42744c4a8858eda48ae8f31f6cdead6479af6a40b2ed009ceed561a7d447ad41"}
Aug 04 22:53:47.961129 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.961036    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:47.988910 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.988875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:47.989402 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:47.989361    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" podStartSLOduration=1.5699479200000002 podStartE2EDuration="1.989346966s" totalImagesPullingTime="419.399046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:53:47.986346671 +0000 UTC m=+1965.527839844" watchObservedRunningTime="2026-08-04 22:53:47.989346966 +0000 UTC m=+1965.530840140"
Aug 04 22:53:48.331026 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:48.330931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:53:48.333707 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:53:48.333679    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f72ad59_b28a_47bd_8e8b_10d9ae9101ea.slice/crio-632a14f11cd606fc75026b4a95245800eed307e412f29c9723de33ff85c2db03 WatchSource:0}: Error finding container 632a14f11cd606fc75026b4a95245800eed307e412f29c9723de33ff85c2db03: Status 404 returned error can't find the container with id 632a14f11cd606fc75026b4a95245800eed307e412f29c9723de33ff85c2db03
Aug 04 22:53:48.968054 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:48.968020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" event={"ID":"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea","Type":"ContainerStarted","Data":"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"}
Aug 04 22:53:48.968501 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:48.968080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" event={"ID":"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea","Type":"ContainerStarted","Data":"632a14f11cd606fc75026b4a95245800eed307e412f29c9723de33ff85c2db03"}
Aug 04 22:53:48.989395 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:48.989345    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" podStartSLOduration=1.989330985 podStartE2EDuration="1.989330985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:53:48.988251012 +0000 UTC m=+1966.529744187" watchObservedRunningTime="2026-08-04 22:53:48.989330985 +0000 UTC m=+1966.530824159"
Aug 04 22:53:49.973273 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:49.973240    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:53:53.974308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:53.974224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:53:54.312413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:54.312334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:53:54.374758 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:53:54.374723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:00.980391 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:00.980356    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:54:17.421439 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.421400    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:54:17.422037 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.421728    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" podUID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" containerName="maas-api" containerID="cri-o://d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a" gracePeriod=30
Aug 04 22:54:17.459965 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.459932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:17.479597 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.479570    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:54:17.479827 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.479798    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" podUID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" containerName="payload-processing" containerID="cri-o://17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212" gracePeriod=30
Aug 04 22:54:17.559339 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.559305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:17.634102 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.634067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:17.709811 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.709778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:17.738273 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.738251    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:54:17.779918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.773669    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:54:17.816050 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.816020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:17.924593 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.924565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxv42\" (UniqueName: \"kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42\") pod \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") "
Aug 04 22:54:17.924749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.924625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn9mw\" (UniqueName: \"kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw\") pod \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") "
Aug 04 22:54:17.924749 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.924652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls\") pod \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") "
Aug 04 22:54:17.924866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.924756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume\") pod \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\" (UID: \"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea\") "
Aug 04 22:54:17.924866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.924796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca\") pod \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\" (UID: \"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed\") "
Aug 04 22:54:17.925206 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.925180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" (UID: "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:54:17.925275 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.925187    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" (UID: "1f72ad59-b28a-47bd-8e8b-10d9ae9101ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:54:17.926934 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.926881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw" (OuterVolumeSpecName: "kube-api-access-rn9mw") pod "1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" (UID: "1f72ad59-b28a-47bd-8e8b-10d9ae9101ea"). InnerVolumeSpecName "kube-api-access-rn9mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:54:17.927027 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.926955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42" (OuterVolumeSpecName: "kube-api-access-wxv42") pod "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" (UID: "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed"). InnerVolumeSpecName "kube-api-access-wxv42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:54:17.927027 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:17.926956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" (UID: "6f16fad7-b0b6-41a0-9628-5d1b2a1964ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:54:18.018299 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.018238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:18.025701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.025678    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn9mw\" (UniqueName: \"kubernetes.io/projected/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-kube-api-access-rn9mw\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:54:18.025766 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.025702    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:54:18.025766 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.025714    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:54:18.025766 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.025733    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:54:18.025766 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.025742    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv42\" (UniqueName: \"kubernetes.io/projected/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed-kube-api-access-wxv42\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:54:18.110234 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.110197    2582 generic.go:356] "Generic (PLEG): container finished" podID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" containerID="17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212" exitCode=0
Aug 04 22:54:18.110413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.110241    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"
Aug 04 22:54:18.110413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.110275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" event={"ID":"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea","Type":"ContainerDied","Data":"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"}
Aug 04 22:54:18.110413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.110337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7" event={"ID":"1f72ad59-b28a-47bd-8e8b-10d9ae9101ea","Type":"ContainerDied","Data":"632a14f11cd606fc75026b4a95245800eed307e412f29c9723de33ff85c2db03"}
Aug 04 22:54:18.110413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.110367    2582 scope.go:117] "RemoveContainer" containerID="17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"
Aug 04 22:54:18.112116 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.112090    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" containerID="d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a" exitCode=0
Aug 04 22:54:18.112211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.112160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" event={"ID":"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed","Type":"ContainerDied","Data":"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"}
Aug 04 22:54:18.112211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.112169    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"
Aug 04 22:54:18.112211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.112204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p" event={"ID":"6f16fad7-b0b6-41a0-9628-5d1b2a1964ed","Type":"ContainerDied","Data":"42744c4a8858eda48ae8f31f6cdead6479af6a40b2ed009ceed561a7d447ad41"}
Aug 04 22:54:18.120982 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.120957    2582 scope.go:117] "RemoveContainer" containerID="17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"
Aug 04 22:54:18.121293 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:54:18.121270    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212\": container with ID starting with 17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212 not found: ID does not exist" containerID="17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"
Aug 04 22:54:18.121378 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.121299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212"} err="failed to get container status \"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212\": rpc error: code = NotFound desc = could not find container \"17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212\": container with ID starting with 17a3986871febded56954fbcf61ca791b10f4f32a8ccba9cb7ef24b2cb068212 not found: ID does not exist"
Aug 04 22:54:18.121378 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.121316    2582 scope.go:117] "RemoveContainer" containerID="d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"
Aug 04 22:54:18.129845 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.129826    2582 scope.go:117] "RemoveContainer" containerID="d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"
Aug 04 22:54:18.130160 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:54:18.130142    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a\": container with ID starting with d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a not found: ID does not exist" containerID="d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"
Aug 04 22:54:18.130212 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.130166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a"} err="failed to get container status \"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a\": rpc error: code = NotFound desc = could not find container \"d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a\": container with ID starting with d46620065503150196518302866719970eb33038c6cf040a8a4503d1cb37082a not found: ID does not exist"
Aug 04 22:54:18.141865 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.141845    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:54:18.143873 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.143846    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63f65db9-d84cccb7b-hgl4p"]
Aug 04 22:54:18.159563 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.159524    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:54:18.161568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.161548    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-63f65db9-65b4968785-qkwz7"]
Aug 04 22:54:18.230252 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.230222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:18.545481 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.545454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:18.660404 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:18.660375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:19.010747 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:19.010721    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" path="/var/lib/kubelet/pods/1f72ad59-b28a-47bd-8e8b-10d9ae9101ea/volumes"
Aug 04 22:54:19.011139 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:19.011124    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" path="/var/lib/kubelet/pods/6f16fad7-b0b6-41a0-9628-5d1b2a1964ed/volumes"
Aug 04 22:54:20.859469 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:20.859429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:20.955304 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:20.955274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:21.960458 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960424    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:54:21.960838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960797    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" containerName="maas-api"
Aug 04 22:54:21.960838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960808    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" containerName="maas-api"
Aug 04 22:54:21.960838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960821    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" containerName="payload-processing"
Aug 04 22:54:21.960838 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960828    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" containerName="payload-processing"
Aug 04 22:54:21.961001 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960889    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f72ad59-b28a-47bd-8e8b-10d9ae9101ea" containerName="payload-processing"
Aug 04 22:54:21.961001 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.960918    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f16fad7-b0b6-41a0-9628-5d1b2a1964ed" containerName="maas-api"
Aug 04 22:54:21.965490 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.965470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:21.968115 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.968099    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9192747d-serving-cert\""
Aug 04 22:54:21.978049 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:21.978025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:54:22.013086 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.013057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:22.159463 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.159431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ck7\" (UniqueName: \"kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.159633 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.159477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.159633 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.159512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.252455 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.252380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:22.260619 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.260600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ck7\" (UniqueName: \"kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.260710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.260641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.260710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.260668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.261358 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.261342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.263308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.263291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.269301 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.269284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ck7\" (UniqueName: \"kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7\") pod \"maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.275491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.275471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:22.413135 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.413106    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:54:22.416436 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:54:22.416406    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db6b1e3_0072_4037_89b1_a11c8cde52bc.slice/crio-efe7ca4c56dff8edeb258d813d29b9edc541274b72aaddd9d060efa3aaa1cbf7 WatchSource:0}: Error finding container efe7ca4c56dff8edeb258d813d29b9edc541274b72aaddd9d060efa3aaa1cbf7: Status 404 returned error can't find the container with id efe7ca4c56dff8edeb258d813d29b9edc541274b72aaddd9d060efa3aaa1cbf7
Aug 04 22:54:22.522438 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.522363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:22.616404 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.616370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:54:22.620599 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.620573    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:54:22.625744 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.625728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.628202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.628186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9192747d-dockercfg-427hr\""
Aug 04 22:54:22.628280 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.628193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9192747d\""
Aug 04 22:54:22.633295 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.633089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:54:22.667491 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.667468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:54:22.670835 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.670818    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.688930 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.688879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:54:22.766655 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.766627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chmt\" (UniqueName: \"kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.766817 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.766671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.867193 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.867172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mgj\" (UniqueName: \"kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.867291 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.867219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chmt\" (UniqueName: \"kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.867291 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.867249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.867291 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.867275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.867868 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.867850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.876085 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.876060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chmt\" (UniqueName: \"kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt\") pod \"payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.938052 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.938031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:22.968380 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.968343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.968724 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.968423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mgj\" (UniqueName: \"kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.969044 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.969025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.976065 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.976043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mgj\" (UniqueName: \"kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj\") pod \"payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") " pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:22.980949 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:22.980881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:23.085849 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:23.085810    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:54:23.088821 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:54:23.088425    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c660f7_1dc4_4330_9398_3d98e1d08c5e.slice/crio-b18233613d91d98fdff39b5002df27ff9b614b9e43a428be985e3048d21cdd74 WatchSource:0}: Error finding container b18233613d91d98fdff39b5002df27ff9b614b9e43a428be985e3048d21cdd74: Status 404 returned error can't find the container with id b18233613d91d98fdff39b5002df27ff9b614b9e43a428be985e3048d21cdd74
Aug 04 22:54:23.138739 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:23.138713    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:54:23.140837 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:54:23.140813    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0387d4cf_6a4f_4043_9dc0_1a1db2032331.slice/crio-d3a0ce4d01497d7c90cc105c846c49978c1fe97bcccfe681e2e281b68bac42c2 WatchSource:0}: Error finding container d3a0ce4d01497d7c90cc105c846c49978c1fe97bcccfe681e2e281b68bac42c2: Status 404 returned error can't find the container with id d3a0ce4d01497d7c90cc105c846c49978c1fe97bcccfe681e2e281b68bac42c2
Aug 04 22:54:23.144829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:23.144803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" event={"ID":"98c660f7-1dc4-4330-9398-3d98e1d08c5e","Type":"ContainerStarted","Data":"b18233613d91d98fdff39b5002df27ff9b614b9e43a428be985e3048d21cdd74"}
Aug 04 22:54:23.146494 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:23.146472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" event={"ID":"6db6b1e3-0072-4037-89b1-a11c8cde52bc","Type":"ContainerStarted","Data":"efe7ca4c56dff8edeb258d813d29b9edc541274b72aaddd9d060efa3aaa1cbf7"}
Aug 04 22:54:24.153109 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.153065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" event={"ID":"6db6b1e3-0072-4037-89b1-a11c8cde52bc","Type":"ContainerStarted","Data":"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"}
Aug 04 22:54:24.153529 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.153217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:24.154710 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.154683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" event={"ID":"0387d4cf-6a4f-4043-9dc0-1a1db2032331","Type":"ContainerStarted","Data":"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"}
Aug 04 22:54:24.154845 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.154736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" event={"ID":"0387d4cf-6a4f-4043-9dc0-1a1db2032331","Type":"ContainerStarted","Data":"d3a0ce4d01497d7c90cc105c846c49978c1fe97bcccfe681e2e281b68bac42c2"}
Aug 04 22:54:24.154845 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.154785    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:24.156187 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.156163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" event={"ID":"98c660f7-1dc4-4330-9398-3d98e1d08c5e","Type":"ContainerStarted","Data":"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"}
Aug 04 22:54:24.156283 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.156206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:54:24.168979 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.168938    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" podStartSLOduration=2.515217355 podStartE2EDuration="3.168925457s" totalImagesPullingTime="653.708102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:54:24.167946424 +0000 UTC m=+2001.709439597" watchObservedRunningTime="2026-08-04 22:54:24.168925457 +0000 UTC m=+2001.710418631"
Aug 04 22:54:24.183293 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.183255    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" podStartSLOduration=2.183242873 podStartE2EDuration="2.183242873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:54:24.181799796 +0000 UTC m=+2001.723292970" watchObservedRunningTime="2026-08-04 22:54:24.183242873 +0000 UTC m=+2001.724736060"
Aug 04 22:54:24.196650 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:24.196611    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" podStartSLOduration=2.196598831 podStartE2EDuration="2.196598831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:54:24.196204714 +0000 UTC m=+2001.737697890" watchObservedRunningTime="2026-08-04 22:54:24.196598831 +0000 UTC m=+2001.738092004"
Aug 04 22:54:30.167237 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:30.167208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:54:35.163701 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:35.163665    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:54:35.164159 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:54:35.163839    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:55:02.605015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.604973    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:55:02.605628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.605261    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" podUID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" containerName="maas-api" containerID="cri-o://6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29" gracePeriod=30
Aug 04 22:55:02.629293 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.629267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:02.661760 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.661720    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:55:02.661964 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.661940    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" podUID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" containerName="payload-processing" containerID="cri-o://d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc" gracePeriod=30
Aug 04 22:55:02.670477 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.670449    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:55:02.670669 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.670647    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" podUID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" containerName="payload-pre-processing" containerID="cri-o://467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71" gracePeriod=30
Aug 04 22:55:02.696113 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.696015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:02.839552 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.839340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:02.923627 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.923595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:02.945581 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.945555    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:55:02.949003 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.948970    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:55:02.961855 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:02.961835    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:55:03.106148 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5chmt\" (UniqueName: \"kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt\") pod \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") "
Aug 04 22:55:03.106352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume\") pod \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") "
Aug 04 22:55:03.106352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls\") pod \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") "
Aug 04 22:55:03.106352 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106328    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2ck7\" (UniqueName: \"kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7\") pod \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") "
Aug 04 22:55:03.106506 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca\") pod \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\" (UID: \"6db6b1e3-0072-4037-89b1-a11c8cde52bc\") "
Aug 04 22:55:03.106506 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume\") pod \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\" (UID: \"98c660f7-1dc4-4330-9398-3d98e1d08c5e\") "
Aug 04 22:55:03.106506 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28mgj\" (UniqueName: \"kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj\") pod \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\" (UID: \"0387d4cf-6a4f-4043-9dc0-1a1db2032331\") "
Aug 04 22:55:03.106643 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106524    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0387d4cf-6a4f-4043-9dc0-1a1db2032331" (UID: "0387d4cf-6a4f-4043-9dc0-1a1db2032331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:55:03.107357 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.106874    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0387d4cf-6a4f-4043-9dc0-1a1db2032331-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.107357 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.107002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6db6b1e3-0072-4037-89b1-a11c8cde52bc" (UID: "6db6b1e3-0072-4037-89b1-a11c8cde52bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:55:03.107357 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.107063    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98c660f7-1dc4-4330-9398-3d98e1d08c5e" (UID: "98c660f7-1dc4-4330-9398-3d98e1d08c5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:55:03.109487 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.109459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt" (OuterVolumeSpecName: "kube-api-access-5chmt") pod "98c660f7-1dc4-4330-9398-3d98e1d08c5e" (UID: "98c660f7-1dc4-4330-9398-3d98e1d08c5e"). InnerVolumeSpecName "kube-api-access-5chmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:55:03.109616 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.109492    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj" (OuterVolumeSpecName: "kube-api-access-28mgj") pod "0387d4cf-6a4f-4043-9dc0-1a1db2032331" (UID: "0387d4cf-6a4f-4043-9dc0-1a1db2032331"). InnerVolumeSpecName "kube-api-access-28mgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:55:03.109616 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.109598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6db6b1e3-0072-4037-89b1-a11c8cde52bc" (UID: "6db6b1e3-0072-4037-89b1-a11c8cde52bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:55:03.110394 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.110367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7" (OuterVolumeSpecName: "kube-api-access-f2ck7") pod "6db6b1e3-0072-4037-89b1-a11c8cde52bc" (UID: "6db6b1e3-0072-4037-89b1-a11c8cde52bc"). InnerVolumeSpecName "kube-api-access-f2ck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:55:03.207848 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207822    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chmt\" (UniqueName: \"kubernetes.io/projected/98c660f7-1dc4-4330-9398-3d98e1d08c5e-kube-api-access-5chmt\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.207848 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207847    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db6b1e3-0072-4037-89b1-a11c8cde52bc-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.208030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207857    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2ck7\" (UniqueName: \"kubernetes.io/projected/6db6b1e3-0072-4037-89b1-a11c8cde52bc-kube-api-access-f2ck7\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.208030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207866    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db6b1e3-0072-4037-89b1-a11c8cde52bc-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.208030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207875    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c660f7-1dc4-4330-9398-3d98e1d08c5e-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.208030 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.207884    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mgj\" (UniqueName: \"kubernetes.io/projected/0387d4cf-6a4f-4043-9dc0-1a1db2032331-kube-api-access-28mgj\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:03.348397 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.348362    2582 generic.go:356] "Generic (PLEG): container finished" podID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" containerID="467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71" exitCode=0
Aug 04 22:55:03.348568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.348412    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"
Aug 04 22:55:03.348568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.348440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" event={"ID":"98c660f7-1dc4-4330-9398-3d98e1d08c5e","Type":"ContainerDied","Data":"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"}
Aug 04 22:55:03.348568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.348490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t" event={"ID":"98c660f7-1dc4-4330-9398-3d98e1d08c5e","Type":"ContainerDied","Data":"b18233613d91d98fdff39b5002df27ff9b614b9e43a428be985e3048d21cdd74"}
Aug 04 22:55:03.348568 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.348514    2582 scope.go:117] "RemoveContainer" containerID="467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"
Aug 04 22:55:03.349977 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.349955    2582 generic.go:356] "Generic (PLEG): container finished" podID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" containerID="6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29" exitCode=0
Aug 04 22:55:03.350099 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.350007    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"
Aug 04 22:55:03.350099 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.350032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" event={"ID":"6db6b1e3-0072-4037-89b1-a11c8cde52bc","Type":"ContainerDied","Data":"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"}
Aug 04 22:55:03.350099 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.350066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8" event={"ID":"6db6b1e3-0072-4037-89b1-a11c8cde52bc","Type":"ContainerDied","Data":"efe7ca4c56dff8edeb258d813d29b9edc541274b72aaddd9d060efa3aaa1cbf7"}
Aug 04 22:55:03.351604 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.351579    2582 generic.go:356] "Generic (PLEG): container finished" podID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" containerID="d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc" exitCode=0
Aug 04 22:55:03.351727 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.351616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" event={"ID":"0387d4cf-6a4f-4043-9dc0-1a1db2032331","Type":"ContainerDied","Data":"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"}
Aug 04 22:55:03.351727 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.351654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6" event={"ID":"0387d4cf-6a4f-4043-9dc0-1a1db2032331","Type":"ContainerDied","Data":"d3a0ce4d01497d7c90cc105c846c49978c1fe97bcccfe681e2e281b68bac42c2"}
Aug 04 22:55:03.351727 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.351620    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"
Aug 04 22:55:03.358593 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.358488    2582 scope.go:117] "RemoveContainer" containerID="467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"
Aug 04 22:55:03.358772 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:55:03.358751    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71\": container with ID starting with 467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71 not found: ID does not exist" containerID="467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"
Aug 04 22:55:03.358853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.358786    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71"} err="failed to get container status \"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71\": rpc error: code = NotFound desc = could not find container \"467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71\": container with ID starting with 467a60ff2255e16f1fb0fdfd7b9e4d817eab6531713c609d5b2e0e74202c8a71 not found: ID does not exist"
Aug 04 22:55:03.358853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.358802    2582 scope.go:117] "RemoveContainer" containerID="6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"
Aug 04 22:55:03.367503 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.367486    2582 scope.go:117] "RemoveContainer" containerID="6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"
Aug 04 22:55:03.367752 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:55:03.367733    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29\": container with ID starting with 6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29 not found: ID does not exist" containerID="6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"
Aug 04 22:55:03.367825 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.367758    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29"} err="failed to get container status \"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29\": rpc error: code = NotFound desc = could not find container \"6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29\": container with ID starting with 6d9cf2f198cfd60b0473cdad20b82a6aec619d77a15cc566d81b31b7b2fd8c29 not found: ID does not exist"
Aug 04 22:55:03.367825 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.367774    2582 scope.go:117] "RemoveContainer" containerID="d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"
Aug 04 22:55:03.372511 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.372451    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:55:03.375983 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.375966    2582 scope.go:117] "RemoveContainer" containerID="d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"
Aug 04 22:55:03.376240 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:55:03.376221    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc\": container with ID starting with d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc not found: ID does not exist" containerID="d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"
Aug 04 22:55:03.376297 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.376245    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc"} err="failed to get container status \"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc\": rpc error: code = NotFound desc = could not find container \"d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc\": container with ID starting with d55c1976a692f1b92932c450ff2a8d7650bde892b77723a4a560cd46b7ed2bbc not found: ID does not exist"
Aug 04 22:55:03.376525 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.376508    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9192747d-7585856f68-8479t"]
Aug 04 22:55:03.386390 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.386332    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:55:03.388561 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.388543    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9192747d-fcf5d4c97-mn6b8"]
Aug 04 22:55:03.398376 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.398354    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:55:03.402204 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:03.402186    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9192747d-76fb45fbdb-nzhb6"]
Aug 04 22:55:04.038490 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:04.038454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:04.260707 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:04.260670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:04.494090 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:04.494041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:04.798659 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:04.798572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:05.011597 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:05.011560    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" path="/var/lib/kubelet/pods/0387d4cf-6a4f-4043-9dc0-1a1db2032331/volumes"
Aug 04 22:55:05.012126 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:05.012103    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" path="/var/lib/kubelet/pods/6db6b1e3-0072-4037-89b1-a11c8cde52bc/volumes"
Aug 04 22:55:05.012595 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:05.012573    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" path="/var/lib/kubelet/pods/98c660f7-1dc4-4330-9398-3d98e1d08c5e/volumes"
Aug 04 22:55:05.429990 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:05.429955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:06.719555 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:06.719518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:06.815519 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:06.815487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:07.802023 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.801882    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:07.802563 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802528    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" containerName="payload-processing"
Aug 04 22:55:07.802563 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802547    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" containerName="payload-processing"
Aug 04 22:55:07.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802584    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" containerName="payload-pre-processing"
Aug 04 22:55:07.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802593    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" containerName="payload-pre-processing"
Aug 04 22:55:07.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802616    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" containerName="maas-api"
Aug 04 22:55:07.802683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802626    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" containerName="maas-api"
Aug 04 22:55:07.802883 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802721    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c660f7-1dc4-4330-9398-3d98e1d08c5e" containerName="payload-pre-processing"
Aug 04 22:55:07.802883 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0387d4cf-6a4f-4043-9dc0-1a1db2032331" containerName="payload-processing"
Aug 04 22:55:07.802883 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.802745    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db6b1e3-0072-4037-89b1-a11c8cde52bc" containerName="maas-api"
Aug 04 22:55:07.806853 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.806833    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:07.809217 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.809197    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8410bf38-serving-cert\""
Aug 04 22:55:07.817829 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.817807    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:07.838612 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.838587    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:07.952102 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.952075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:07.952211 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.952132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmpc\" (UniqueName: \"kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:07.952254 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:07.952210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.034463 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.034439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:08.053056 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.053003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.053134 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.053055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.053134 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.053096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmpc\" (UniqueName: \"kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.053641 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.053620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.055532 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.055508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.062111 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.062086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmpc\" (UniqueName: \"kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc\") pod \"maas-api-e2e-derive-8410bf38-555cc959-z8h4c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.118341 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.118311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:08.250128 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.250103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:08.252721 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:55:08.252689    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b95b1b_ccae_4367_ab81_42c09ffb976c.slice/crio-fe083deade4a478d9e0c14115dddd610611f32f52722497f526cd5f98dde4fca WatchSource:0}: Error finding container fe083deade4a478d9e0c14115dddd610611f32f52722497f526cd5f98dde4fca: Status 404 returned error can't find the container with id fe083deade4a478d9e0c14115dddd610611f32f52722497f526cd5f98dde4fca
Aug 04 22:55:08.377938 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.377881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:08.381413 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.381389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" event={"ID":"d0b95b1b-ccae-4367-ab81-42c09ffb976c","Type":"ContainerStarted","Data":"fe083deade4a478d9e0c14115dddd610611f32f52722497f526cd5f98dde4fca"}
Aug 04 22:55:08.466305 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.466273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:08.470479 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.470456    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:08.475088 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.475073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.477695 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.477669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8410bf38\""
Aug 04 22:55:08.478575 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.478552    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8410bf38-dockercfg-6nnbt\""
Aug 04 22:55:08.486577 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.486555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:08.558011 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.557974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.558170 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.558099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbfc\" (UniqueName: \"kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.659350 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.659320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.659483 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.659390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbfc\" (UniqueName: \"kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.660263 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.660241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.666879 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.666853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbfc\" (UniqueName: \"kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc\") pod \"payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") " pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.786308 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.786276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:08.923038 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:08.922808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:08.925684 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:55:08.925650    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e17491_c0b7_496c_b15f_c3ebfbbf06a4.slice/crio-89a59f2bc8575cc5b407be0fe9b4ce67d948594362fdde1d7c23e4a69e18e682 WatchSource:0}: Error finding container 89a59f2bc8575cc5b407be0fe9b4ce67d948594362fdde1d7c23e4a69e18e682: Status 404 returned error can't find the container with id 89a59f2bc8575cc5b407be0fe9b4ce67d948594362fdde1d7c23e4a69e18e682
Aug 04 22:55:09.387768 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.387731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" event={"ID":"d0b95b1b-ccae-4367-ab81-42c09ffb976c","Type":"ContainerStarted","Data":"60b3aff85d7f8c3fd5f2a1c2a2dc164737eefea972e030fac7ac83211ef7d996"}
Aug 04 22:55:09.387985 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.387813    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:09.389335 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.389312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" event={"ID":"35e17491-c0b7-496c-b15f-c3ebfbbf06a4","Type":"ContainerStarted","Data":"e485affa4ccb740ef7ad0cedeb09489c92b168ba4617df9c70544af8c9d6bee2"}
Aug 04 22:55:09.389440 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.389350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" event={"ID":"35e17491-c0b7-496c-b15f-c3ebfbbf06a4","Type":"ContainerStarted","Data":"89a59f2bc8575cc5b407be0fe9b4ce67d948594362fdde1d7c23e4a69e18e682"}
Aug 04 22:55:09.389478 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.389443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:09.405557 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.405505    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" podStartSLOduration=2.017219282 podStartE2EDuration="2.40549294s" totalImagesPullingTime="388.273658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:55:09.404375175 +0000 UTC m=+2046.945868362" watchObservedRunningTime="2026-08-04 22:55:09.40549294 +0000 UTC m=+2046.946986144"
Aug 04 22:55:09.419570 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:09.419521    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" podStartSLOduration=1.419503295 podStartE2EDuration="1.419503295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:55:09.418740795 +0000 UTC m=+2046.960233984" watchObservedRunningTime="2026-08-04 22:55:09.419503295 +0000 UTC m=+2046.960996470"
Aug 04 22:55:11.499047 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:11.499017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:12.045097 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:12.045069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:12.146204 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:12.146171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:15.398367 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:15.398334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:20.395812 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:20.395783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:29.928046 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:29.927970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.340512 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.340472    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:38.341025 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.340694    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" podUID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" containerName="maas-api" containerID="cri-o://60b3aff85d7f8c3fd5f2a1c2a2dc164737eefea972e030fac7ac83211ef7d996" gracePeriod=30
Aug 04 22:55:38.370951 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.370925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.394963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.394935    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:38.395158 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.395133    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" podUID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" containerName="payload-processing" containerID="cri-o://e485affa4ccb740ef7ad0cedeb09489c92b168ba4617df9c70544af8c9d6bee2" gracePeriod=30
Aug 04 22:55:38.475095 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.475059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.538628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.538564    2582 generic.go:356] "Generic (PLEG): container finished" podID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" containerID="60b3aff85d7f8c3fd5f2a1c2a2dc164737eefea972e030fac7ac83211ef7d996" exitCode=0
Aug 04 22:55:38.538628 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.538599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" event={"ID":"d0b95b1b-ccae-4367-ab81-42c09ffb976c","Type":"ContainerDied","Data":"60b3aff85d7f8c3fd5f2a1c2a2dc164737eefea972e030fac7ac83211ef7d996"}
Aug 04 22:55:38.540448 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.540424    2582 generic.go:356] "Generic (PLEG): container finished" podID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" containerID="e485affa4ccb740ef7ad0cedeb09489c92b168ba4617df9c70544af8c9d6bee2" exitCode=0
Aug 04 22:55:38.540588 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.540476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" event={"ID":"35e17491-c0b7-496c-b15f-c3ebfbbf06a4","Type":"ContainerDied","Data":"e485affa4ccb740ef7ad0cedeb09489c92b168ba4617df9c70544af8c9d6bee2"}
Aug 04 22:55:38.559665 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.559628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.647888 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.647859    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:38.674526 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.674497    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:38.684015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.683992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.726963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.724952    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca\") pod \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") "
Aug 04 22:55:38.726963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.725062    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls\") pod \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") "
Aug 04 22:55:38.726963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.725103    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmpc\" (UniqueName: \"kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc\") pod \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\" (UID: \"d0b95b1b-ccae-4367-ab81-42c09ffb976c\") "
Aug 04 22:55:38.726963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.726405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0b95b1b-ccae-4367-ab81-42c09ffb976c" (UID: "d0b95b1b-ccae-4367-ab81-42c09ffb976c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:55:38.733919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.729405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc" (OuterVolumeSpecName: "kube-api-access-jfmpc") pod "d0b95b1b-ccae-4367-ab81-42c09ffb976c" (UID: "d0b95b1b-ccae-4367-ab81-42c09ffb976c"). InnerVolumeSpecName "kube-api-access-jfmpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:55:38.733919 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.730514    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0b95b1b-ccae-4367-ab81-42c09ffb976c" (UID: "d0b95b1b-ccae-4367-ab81-42c09ffb976c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:55:38.759971 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.759940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:38.826221 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume\") pod \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") "
Aug 04 22:55:38.826382 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826233    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdbfc\" (UniqueName: \"kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc\") pod \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\" (UID: \"35e17491-c0b7-496c-b15f-c3ebfbbf06a4\") "
Aug 04 22:55:38.826451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826407    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0b95b1b-ccae-4367-ab81-42c09ffb976c-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:38.826451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826417    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmpc\" (UniqueName: \"kubernetes.io/projected/d0b95b1b-ccae-4367-ab81-42c09ffb976c-kube-api-access-jfmpc\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:38.826451 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826426    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b95b1b-ccae-4367-ab81-42c09ffb976c-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:38.826619 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.826499    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35e17491-c0b7-496c-b15f-c3ebfbbf06a4" (UID: "35e17491-c0b7-496c-b15f-c3ebfbbf06a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:55:38.828246 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.828224    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc" (OuterVolumeSpecName: "kube-api-access-mdbfc") pod "35e17491-c0b7-496c-b15f-c3ebfbbf06a4" (UID: "35e17491-c0b7-496c-b15f-c3ebfbbf06a4"). InnerVolumeSpecName "kube-api-access-mdbfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:55:38.927097 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.927064    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:38.927097 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.927093    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdbfc\" (UniqueName: \"kubernetes.io/projected/35e17491-c0b7-496c-b15f-c3ebfbbf06a4-kube-api-access-mdbfc\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:55:38.950008 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:38.949978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:39.401142 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.401105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:39.501428 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.501395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:39.547266 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.547233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c" event={"ID":"d0b95b1b-ccae-4367-ab81-42c09ffb976c","Type":"ContainerDied","Data":"fe083deade4a478d9e0c14115dddd610611f32f52722497f526cd5f98dde4fca"}
Aug 04 22:55:39.547431 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.547303    2582 scope.go:117] "RemoveContainer" containerID="60b3aff85d7f8c3fd5f2a1c2a2dc164737eefea972e030fac7ac83211ef7d996"
Aug 04 22:55:39.547431 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.547246    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"
Aug 04 22:55:39.548777 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.548760    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"
Aug 04 22:55:39.552913 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.548794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn" event={"ID":"35e17491-c0b7-496c-b15f-c3ebfbbf06a4","Type":"ContainerDied","Data":"89a59f2bc8575cc5b407be0fe9b4ce67d948594362fdde1d7c23e4a69e18e682"}
Aug 04 22:55:39.561600 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.561580    2582 scope.go:117] "RemoveContainer" containerID="e485affa4ccb740ef7ad0cedeb09489c92b168ba4617df9c70544af8c9d6bee2"
Aug 04 22:55:39.567719 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.567699    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:39.571662 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.571639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:39.573823 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.573802    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8410bf38-555cc959-z8h4c"]
Aug 04 22:55:39.589831 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.589805    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:39.593112 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:39.593091    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8410bf38-5bb5fb565d-rk5qn"]
Aug 04 22:55:41.010551 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:41.010519    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" path="/var/lib/kubelet/pods/35e17491-c0b7-496c-b15f-c3ebfbbf06a4/volumes"
Aug 04 22:55:41.010963 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:41.010885    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" path="/var/lib/kubelet/pods/d0b95b1b-ccae-4367-ab81-42c09ffb976c/volumes"
Aug 04 22:55:51.234571 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:51.234529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:57.873683 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:57.873639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:57.975197 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:57.975166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:58.075578 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:58.075543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:58.177138 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:58.177048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:55:58.575396 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:55:58.575298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:03.036176 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:03.036141    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:56:03.041610 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:03.041588    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 22:56:24.578191 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:24.578135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:30.842444 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:30.842405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:30.954498 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:30.954450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:31.047044 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:31.047015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:45.325121 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:45.325079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:46.973861 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:46.973825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:52.668168 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:52.668120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:52.737333 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:52.737305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:52.845477 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:52.845446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:56:52.942160 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:56:52.942088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:03.393236 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:03.393198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:14.427918 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:14.427859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:20.474058 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:20.474008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:20.579559 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:20.579531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:20.675814 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:20.675784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:20.733756 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:20.733685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:21.413294 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:21.413257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:21.513347 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:21.513299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:21.623567 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:21.623532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:23.147028 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:23.146987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:29.731940 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:29.731869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:29.832023 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:29.831989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:30.411202 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:30.411137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:36.912781 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:36.912743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:37.014307 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:37.014266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:37.158631 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:37.158596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:37.479396 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:37.479363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:37.770089 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:37.769999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:40.233313 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:40.233281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:40.330729 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:40.330696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:41.321627 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.321594    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:57:41.322028 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322013    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" containerName="payload-processing"
Aug 04 22:57:41.322083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322030    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" containerName="payload-processing"
Aug 04 22:57:41.322083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322048    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" containerName="maas-api"
Aug 04 22:57:41.322083 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322053    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" containerName="maas-api"
Aug 04 22:57:41.322223 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322118    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e17491-c0b7-496c-b15f-c3ebfbbf06a4" containerName="payload-processing"
Aug 04 22:57:41.322223 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.322127    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b95b1b-ccae-4367-ab81-42c09ffb976c" containerName="maas-api"
Aug 04 22:57:41.325386 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.325370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.327865 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.327844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3c68233e-serving-cert\""
Aug 04 22:57:41.334318 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.334297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:57:41.348754 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.348729    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:41.494282 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.494256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.494392 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.494375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98fg\" (UniqueName: \"kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.494437 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.494403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.595695 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.595636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98fg\" (UniqueName: \"kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.595695 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.595669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.595844 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.595701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.595844 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:57:41.595781    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-serving-cert: secret "maas-api-e2e-mt-3c68233e-serving-cert" not found
Aug 04 22:57:41.595962 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:57:41.595860    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls podName:53bae1d9-1443-4c76-a3ce-4a279ce0e866 nodeName:}" failed. No retries permitted until 2026-08-04 22:57:42.095844275 +0000 UTC m=+2199.637337431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls") pod "maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" (UID: "53bae1d9-1443-4c76-a3ce-4a279ce0e866") : secret "maas-api-e2e-mt-3c68233e-serving-cert" not found
Aug 04 22:57:41.596302 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.596286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.604088 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.604057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98fg\" (UniqueName: \"kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:41.956271 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:41.956232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:42.059078 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.059047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:42.099622 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.099590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:42.102232 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.102214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") pod \"maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:42.112457 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.112433    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:57:42.116112 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.116098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.123220 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.123195    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3c68233e-dockercfg-86khd\""
Aug 04 22:57:42.123404 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.123379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3c68233e\""
Aug 04 22:57:42.130234 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.130211    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:57:42.150418 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.150391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:57:42.200705 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.200678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.200819 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.200723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzdt\" (UniqueName: \"kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.236987 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.236905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:42.301605 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.301560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.301776 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.301626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzdt\" (UniqueName: \"kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.302442 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.302418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.309930 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.309882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzdt\" (UniqueName: \"kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt\") pod \"payload-processing-e2e-mt-3c68233e-b4b987554-g75dt\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") " pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.374357 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.374331    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:57:42.376339 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:57:42.376312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bae1d9_1443_4c76_a3ce_4a279ce0e866.slice/crio-78e055d71390113b75e0fda0772de154ef9a3557afabcc600a03f3636822b3ae WatchSource:0}: Error finding container 78e055d71390113b75e0fda0772de154ef9a3557afabcc600a03f3636822b3ae: Status 404 returned error can't find the container with id 78e055d71390113b75e0fda0772de154ef9a3557afabcc600a03f3636822b3ae
Aug 04 22:57:42.426824 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.426800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:42.768067 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:42.768039    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:57:42.772493 ip-10-0-142-146 kubenswrapper[2582]: W0804 22:57:42.770007    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f296913_4be4_4469_a8e7_e1bc13f2e6da.slice/crio-b5ab600f226999be360c411c816908196ce078db3e0dfc8ce229d24a1075414a WatchSource:0}: Error finding container b5ab600f226999be360c411c816908196ce078db3e0dfc8ce229d24a1075414a: Status 404 returned error can't find the container with id b5ab600f226999be360c411c816908196ce078db3e0dfc8ce229d24a1075414a
Aug 04 22:57:43.144990 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.144955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" event={"ID":"9f296913-4be4-4469-a8e7-e1bc13f2e6da","Type":"ContainerStarted","Data":"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"}
Aug 04 22:57:43.145168 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.145005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" event={"ID":"9f296913-4be4-4469-a8e7-e1bc13f2e6da","Type":"ContainerStarted","Data":"b5ab600f226999be360c411c816908196ce078db3e0dfc8ce229d24a1075414a"}
Aug 04 22:57:43.145168 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.145118    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:57:43.146539 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.146517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" event={"ID":"53bae1d9-1443-4c76-a3ce-4a279ce0e866","Type":"ContainerStarted","Data":"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"}
Aug 04 22:57:43.146659 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.146571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" event={"ID":"53bae1d9-1443-4c76-a3ce-4a279ce0e866","Type":"ContainerStarted","Data":"78e055d71390113b75e0fda0772de154ef9a3557afabcc600a03f3636822b3ae"}
Aug 04 22:57:43.146659 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.146601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:43.161085 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.161042    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" podStartSLOduration=1.161029273 podStartE2EDuration="1.161029273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:57:43.15974691 +0000 UTC m=+2200.701240112" watchObservedRunningTime="2026-08-04 22:57:43.161029273 +0000 UTC m=+2200.702522447"
Aug 04 22:57:43.182346 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:43.182301    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" podStartSLOduration=1.713787559 podStartE2EDuration="2.182288622s" totalImagesPullingTime="468.501063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 22:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 22:57:43.180756616 +0000 UTC m=+2200.722249787" watchObservedRunningTime="2026-08-04 22:57:43.182288622 +0000 UTC m=+2200.723781796"
Aug 04 22:57:49.156984 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:49.156958    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:57:54.154215 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:57:54.154182    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:58:05.204094 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:05.204065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:05.910338 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:05.910296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:05.998907 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:05.998867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:06.337856 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:06.337777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:12.696649 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:12.696608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:12.763258 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:12.763227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:16.567579 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.567544    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:58:16.568135 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.567753    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" podUID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" containerName="maas-api" containerID="cri-o://64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa" gracePeriod=30
Aug 04 22:58:16.596311 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.596286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:16.630795 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.630767    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:58:16.631061 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.631036    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" podUID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" containerName="payload-processing" containerID="cri-o://f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8" gracePeriod=30
Aug 04 22:58:16.674605 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.674578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:16.779581 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.779544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:16.837015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.836986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:16.905408 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.905379    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:58:16.909090 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.909068    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:58:16.990465 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume\") pod \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") "
Aug 04 22:58:16.990598 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca\") pod \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") "
Aug 04 22:58:16.990598 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hzdt\" (UniqueName: \"kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt\") pod \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\" (UID: \"9f296913-4be4-4469-a8e7-e1bc13f2e6da\") "
Aug 04 22:58:16.990598 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990581    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") pod \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") "
Aug 04 22:58:16.990736 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g98fg\" (UniqueName: \"kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg\") pod \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\" (UID: \"53bae1d9-1443-4c76-a3ce-4a279ce0e866\") "
Aug 04 22:58:16.990848 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990826    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f296913-4be4-4469-a8e7-e1bc13f2e6da" (UID: "9f296913-4be4-4469-a8e7-e1bc13f2e6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:58:16.990941 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53bae1d9-1443-4c76-a3ce-4a279ce0e866" (UID: "53bae1d9-1443-4c76-a3ce-4a279ce0e866"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 22:58:16.991010 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.990956    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f296913-4be4-4469-a8e7-e1bc13f2e6da-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:58:16.992815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.992783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg" (OuterVolumeSpecName: "kube-api-access-g98fg") pod "53bae1d9-1443-4c76-a3ce-4a279ce0e866" (UID: "53bae1d9-1443-4c76-a3ce-4a279ce0e866"). InnerVolumeSpecName "kube-api-access-g98fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:58:16.992815 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.992797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53bae1d9-1443-4c76-a3ce-4a279ce0e866" (UID: "53bae1d9-1443-4c76-a3ce-4a279ce0e866"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 22:58:16.992966 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:16.992817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt" (OuterVolumeSpecName: "kube-api-access-8hzdt") pod "9f296913-4be4-4469-a8e7-e1bc13f2e6da" (UID: "9f296913-4be4-4469-a8e7-e1bc13f2e6da"). InnerVolumeSpecName "kube-api-access-8hzdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 22:58:17.092344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.092277    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53bae1d9-1443-4c76-a3ce-4a279ce0e866-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:58:17.092344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.092301    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzdt\" (UniqueName: \"kubernetes.io/projected/9f296913-4be4-4469-a8e7-e1bc13f2e6da-kube-api-access-8hzdt\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:58:17.092344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.092311    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53bae1d9-1443-4c76-a3ce-4a279ce0e866-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:58:17.092344 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.092319    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98fg\" (UniqueName: \"kubernetes.io/projected/53bae1d9-1443-4c76-a3ce-4a279ce0e866-kube-api-access-g98fg\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 22:58:17.169432 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.169401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:17.317015 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.316983    2582 generic.go:356] "Generic (PLEG): container finished" podID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" containerID="64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa" exitCode=0
Aug 04 22:58:17.317203 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.317046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"
Aug 04 22:58:17.317203 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.317074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" event={"ID":"53bae1d9-1443-4c76-a3ce-4a279ce0e866","Type":"ContainerDied","Data":"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"}
Aug 04 22:58:17.317203 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.317149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh" event={"ID":"53bae1d9-1443-4c76-a3ce-4a279ce0e866","Type":"ContainerDied","Data":"78e055d71390113b75e0fda0772de154ef9a3557afabcc600a03f3636822b3ae"}
Aug 04 22:58:17.317203 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.317182    2582 scope.go:117] "RemoveContainer" containerID="64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"
Aug 04 22:58:17.318686 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.318618    2582 generic.go:356] "Generic (PLEG): container finished" podID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" containerID="f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8" exitCode=0
Aug 04 22:58:17.318686 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.318644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" event={"ID":"9f296913-4be4-4469-a8e7-e1bc13f2e6da","Type":"ContainerDied","Data":"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"}
Aug 04 22:58:17.318686 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.318680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt" event={"ID":"9f296913-4be4-4469-a8e7-e1bc13f2e6da","Type":"ContainerDied","Data":"b5ab600f226999be360c411c816908196ce078db3e0dfc8ce229d24a1075414a"}
Aug 04 22:58:17.318686 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.318686    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"
Aug 04 22:58:17.325851 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.325833    2582 scope.go:117] "RemoveContainer" containerID="64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"
Aug 04 22:58:17.326137 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:58:17.326114    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa\": container with ID starting with 64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa not found: ID does not exist" containerID="64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"
Aug 04 22:58:17.326192 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.326148    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa"} err="failed to get container status \"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa\": rpc error: code = NotFound desc = could not find container \"64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa\": container with ID starting with 64b75603937ea3fb9a8e0fba307db06d3405ca07e35ff7185a719648d133baaa not found: ID does not exist"
Aug 04 22:58:17.326192 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.326169    2582 scope.go:117] "RemoveContainer" containerID="f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"
Aug 04 22:58:17.333866 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.333826    2582 scope.go:117] "RemoveContainer" containerID="f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"
Aug 04 22:58:17.334186 ip-10-0-142-146 kubenswrapper[2582]: E0804 22:58:17.334165    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8\": container with ID starting with f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8 not found: ID does not exist" containerID="f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"
Aug 04 22:58:17.334265 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.334193    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8"} err="failed to get container status \"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8\": rpc error: code = NotFound desc = could not find container \"f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8\": container with ID starting with f0133330c3b0154195e74faee583ebe9d8fc56e8648f5cdc8c790065fe82ffe8 not found: ID does not exist"
Aug 04 22:58:17.335998 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.335976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:58:17.340962 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.340939    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3c68233e-6d44d5cc8d-bflsh"]
Aug 04 22:58:17.350415 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.350369    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:58:17.354706 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:17.354687    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3c68233e-b4b987554-g75dt"]
Aug 04 22:58:19.011460 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:19.011433    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" path="/var/lib/kubelet/pods/53bae1d9-1443-4c76-a3ce-4a279ce0e866/volumes"
Aug 04 22:58:19.011813 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:19.011780    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" path="/var/lib/kubelet/pods/9f296913-4be4-4469-a8e7-e1bc13f2e6da/volumes"
Aug 04 22:58:22.874577 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:22.874544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:22.973252 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:22.973189    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:23.136201 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:23.136122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:23.449074 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:23.448991    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:23.581534 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:23.581504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:23.681615 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:23.681582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:23.782329 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:23.782264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:29.619993 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:29.619922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:29.726712 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:29.726646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:29.821268 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:29.821235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:29.952231 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:29.952151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:35.600977 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:35.600942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 22:58:35.701475 ip-10-0-142-146 kubenswrapper[2582]: I0804 22:58:35.701446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:38.199442 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:38.199408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:40.314285 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:40.314224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:41.866174 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:41.866135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:41.962139 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:41.962103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:42.940656 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.940621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:00:42.941275 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941259    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" containerName="payload-processing"
Aug 04 23:00:42.941321 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" containerName="payload-processing"
Aug 04 23:00:42.941321 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941297    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" containerName="maas-api"
Aug 04 23:00:42.941321 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941305    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" containerName="maas-api"
Aug 04 23:00:42.941422 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941379    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f296913-4be4-4469-a8e7-e1bc13f2e6da" containerName="payload-processing"
Aug 04 23:00:42.941422 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.941394    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bae1d9-1443-4c76-a3ce-4a279ce0e866" containerName="maas-api"
Aug 04 23:00:42.944672 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.944645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:42.946931 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.946889    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e083a768-serving-cert\""
Aug 04 23:00:42.954871 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.954849    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:00:42.977544 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:42.977516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:43.041541 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.041513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddzv\" (UniqueName: \"kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.041667 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.041563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.041711 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.041657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.142280 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.142249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.142417 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.142316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddzv\" (UniqueName: \"kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.142417 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.142319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:43.142417 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.142359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.142908 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.142873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.144887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.144866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.150234 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.150210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddzv\" (UniqueName: \"kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv\") pod \"maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.257004 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.256943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:43.390019 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.389993    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:00:43.393221 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:00:43.393192    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89adbd4d_061c_4114_bf7d_bda26556d116.slice/crio-e44d6c2eef823ca367cb0af6dd971558414deca1852c5004277bd78edc60cd62 WatchSource:0}: Error finding container e44d6c2eef823ca367cb0af6dd971558414deca1852c5004277bd78edc60cd62: Status 404 returned error can't find the container with id e44d6c2eef823ca367cb0af6dd971558414deca1852c5004277bd78edc60cd62
Aug 04 23:00:43.394701 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.394684    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 23:00:43.470446 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.470419    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:43.577725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.577647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:00:43.578289 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.578260    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:00:43.582854 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.582839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.585099 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.585078    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e083a768\""
Aug 04 23:00:43.585579 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.585565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e083a768-dockercfg-hwdf6\""
Aug 04 23:00:43.593679 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.593657    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:00:43.610292 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.610267    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:00:43.613545 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.613530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.631455 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.631430    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:00:43.646796 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.646761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2cm\" (UniqueName: \"kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.646954 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.646811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.646954 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.646868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.646954 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.646924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpzg\" (UniqueName: \"kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.748360 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.748331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpzg\" (UniqueName: \"kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.748484 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.748411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2cm\" (UniqueName: \"kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.748484 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.748450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.748600 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.748506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.749222 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.749202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.749222 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.749214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.757340 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.757318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2cm\" (UniqueName: \"kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm\") pod \"payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.757497 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.757477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpzg\" (UniqueName: \"kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg\") pod \"payload-processing-e2e-mt-e083a768-66b8985754-v7xwn\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") " pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:43.894123 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.894098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:43.923576 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:43.923546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:44.038268 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.038222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" event={"ID":"89adbd4d-061c-4114-bf7d-bda26556d116","Type":"ContainerStarted","Data":"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"}
Aug 04 23:00:44.038719 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.038292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" event={"ID":"89adbd4d-061c-4114-bf7d-bda26556d116","Type":"ContainerStarted","Data":"e44d6c2eef823ca367cb0af6dd971558414deca1852c5004277bd78edc60cd62"}
Aug 04 23:00:44.038719 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.038344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:44.050225 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.050192    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:00:44.051649 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:00:44.051619    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e51ded_f193_480e_ad32_3fd353bac78b.slice/crio-101d9130a51b943cc4ce045475326f5d7fbd94208e1034fcdd99ed755bea043b WatchSource:0}: Error finding container 101d9130a51b943cc4ce045475326f5d7fbd94208e1034fcdd99ed755bea043b: Status 404 returned error can't find the container with id 101d9130a51b943cc4ce045475326f5d7fbd94208e1034fcdd99ed755bea043b
Aug 04 23:00:44.056804 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.056749    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" podStartSLOduration=1.531717615 podStartE2EDuration="2.05672847s" totalImagesPullingTime="525.010855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:00:44.055935986 +0000 UTC m=+2381.597429162" watchObservedRunningTime="2026-08-04 23:00:44.05672847 +0000 UTC m=+2381.598221645"
Aug 04 23:00:44.086283 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:44.086258    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:00:44.090036 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:00:44.090011    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4804e0d9_975f_4039_801a_814dcf40fe3a.slice/crio-8255a5a3ca7443a7ce7659a8bedbe432b74aa7a9f16cd9e147b169bedc8ab06e WatchSource:0}: Error finding container 8255a5a3ca7443a7ce7659a8bedbe432b74aa7a9f16cd9e147b169bedc8ab06e: Status 404 returned error can't find the container with id 8255a5a3ca7443a7ce7659a8bedbe432b74aa7a9f16cd9e147b169bedc8ab06e
Aug 04 23:00:45.044655 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.044619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" event={"ID":"c2e51ded-f193-480e-ad32-3fd353bac78b","Type":"ContainerStarted","Data":"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"}
Aug 04 23:00:45.045108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.044673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" event={"ID":"c2e51ded-f193-480e-ad32-3fd353bac78b","Type":"ContainerStarted","Data":"101d9130a51b943cc4ce045475326f5d7fbd94208e1034fcdd99ed755bea043b"}
Aug 04 23:00:45.045108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.044712    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:45.046365 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.046339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" event={"ID":"4804e0d9-975f-4039-801a-814dcf40fe3a","Type":"ContainerStarted","Data":"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"}
Aug 04 23:00:45.046434 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.046392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" event={"ID":"4804e0d9-975f-4039-801a-814dcf40fe3a","Type":"ContainerStarted","Data":"8255a5a3ca7443a7ce7659a8bedbe432b74aa7a9f16cd9e147b169bedc8ab06e"}
Aug 04 23:00:45.046527 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.046514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:00:45.063462 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.063417    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" podStartSLOduration=2.063403068 podStartE2EDuration="2.063403068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:00:45.061942131 +0000 UTC m=+2382.603435297" watchObservedRunningTime="2026-08-04 23:00:45.063403068 +0000 UTC m=+2382.604896242"
Aug 04 23:00:45.081732 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:45.081692    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" podStartSLOduration=2.081679994 podStartE2EDuration="2.081679994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:00:45.080824159 +0000 UTC m=+2382.622317336" watchObservedRunningTime="2026-08-04 23:00:45.081679994 +0000 UTC m=+2382.623173167"
Aug 04 23:00:50.052221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:50.052187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:00:56.054023 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:56.053991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:00:56.054389 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:00:56.054162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:01:03.068544 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:03.068518    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:01:03.073584 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:03.073564    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:01:06.751693 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:06.751660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:01:12.793870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.793838    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:01:12.797407 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.797391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.800611 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.800586    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e083a768\"/\"default-dockercfg-fpwm9\""
Aug 04 23:01:12.800611 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.800599    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e083a768\"/\"kube-root-ca.crt\""
Aug 04 23:01:12.800798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.800660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e083a768\"/\"openshift-service-ca.crt\""
Aug 04 23:01:12.800798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.800586    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e083a768\"/\"e2e-lifecycle-model-e083a768-kserve-self-signed-certs\""
Aug 04 23:01:12.804830 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.804811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:01:12.827000 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.826975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:01:12.895132 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.895280 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.895280 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pnq\" (UniqueName: \"kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.895280 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.895280 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.895474 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.895377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996632 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996632 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pnq\" (UniqueName: \"kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.996856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.996789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.997166 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.997141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.997223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.997191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.997223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.997204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.999100 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.999080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:12.999273 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:12.999257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:13.004300 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:13.004278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pnq\" (UniqueName: \"kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq\") pod \"e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") " pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:13.108461 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:13.108402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:13.239822 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:13.239793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:01:13.241776 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:01:13.241731    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda73f91fc_83e3_44c4_94e7_8004219110be.slice/crio-8cabe1cfd4e80da2015a12f2c26d153ca67cf9262b1d5fe93f2d5555186957eb WatchSource:0}: Error finding container 8cabe1cfd4e80da2015a12f2c26d153ca67cf9262b1d5fe93f2d5555186957eb: Status 404 returned error can't find the container with id 8cabe1cfd4e80da2015a12f2c26d153ca67cf9262b1d5fe93f2d5555186957eb
Aug 04 23:01:14.188841 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:14.188809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" event={"ID":"a73f91fc-83e3-44c4-94e7-8004219110be","Type":"ContainerStarted","Data":"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"}
Aug 04 23:01:14.189254 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:14.188851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" event={"ID":"a73f91fc-83e3-44c4-94e7-8004219110be","Type":"ContainerStarted","Data":"8cabe1cfd4e80da2015a12f2c26d153ca67cf9262b1d5fe93f2d5555186957eb"}
Aug 04 23:01:14.205558 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:14.205516    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" podStartSLOduration=2.2054983 podStartE2EDuration="2.2054983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:01:14.204528457 +0000 UTC m=+2411.746021630" watchObservedRunningTime="2026-08-04 23:01:14.2054983 +0000 UTC m=+2411.746991471"
Aug 04 23:01:22.955459 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:22.955423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:01:23.011398 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:23.011316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:01:23.108938 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:23.108889    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:23.109082 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:23.108973    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:23.121521 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:23.121495    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:23.242581 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:23.242513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:01:51.181103 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:51.181064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:01:59.198131 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:01:59.198090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:01.877745 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.877704    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:02:01.878262 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.877985    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" podUID="89adbd4d-061c-4114-bf7d-bda26556d116" containerName="maas-api" containerID="cri-o://a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64" gracePeriod=30
Aug 04 23:02:01.931535 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.931495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:02:01.931772 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.931745    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" podUID="4804e0d9-975f-4039-801a-814dcf40fe3a" containerName="payload-processing" containerID="cri-o://0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785" gracePeriod=30
Aug 04 23:02:01.939077 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.939046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:02:01.939353 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.939279    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" podUID="c2e51ded-f193-480e-ad32-3fd353bac78b" containerName="payload-pre-processing" containerID="cri-o://3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b" gracePeriod=30
Aug 04 23:02:01.957066 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:01.957039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:02.059369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.059323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:02.213981 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.213950    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:02:02.217427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.217409    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:02:02.241777 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.241754    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:02:02.261151 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.261121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:02.342175 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342137    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng2cm\" (UniqueName: \"kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm\") pod \"c2e51ded-f193-480e-ad32-3fd353bac78b\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") "
Aug 04 23:02:02.342369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342195    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume\") pod \"4804e0d9-975f-4039-801a-814dcf40fe3a\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") "
Aug 04 23:02:02.342369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls\") pod \"89adbd4d-061c-4114-bf7d-bda26556d116\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") "
Aug 04 23:02:02.342369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume\") pod \"c2e51ded-f193-480e-ad32-3fd353bac78b\" (UID: \"c2e51ded-f193-480e-ad32-3fd353bac78b\") "
Aug 04 23:02:02.342369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nddzv\" (UniqueName: \"kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv\") pod \"89adbd4d-061c-4114-bf7d-bda26556d116\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") "
Aug 04 23:02:02.342887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca\") pod \"89adbd4d-061c-4114-bf7d-bda26556d116\" (UID: \"89adbd4d-061c-4114-bf7d-bda26556d116\") "
Aug 04 23:02:02.342887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvpzg\" (UniqueName: \"kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg\") pod \"4804e0d9-975f-4039-801a-814dcf40fe3a\" (UID: \"4804e0d9-975f-4039-801a-814dcf40fe3a\") "
Aug 04 23:02:02.342887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342732    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2e51ded-f193-480e-ad32-3fd353bac78b" (UID: "c2e51ded-f193-480e-ad32-3fd353bac78b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:02:02.342887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4804e0d9-975f-4039-801a-814dcf40fe3a" (UID: "4804e0d9-975f-4039-801a-814dcf40fe3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:02:02.343316 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.342955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89adbd4d-061c-4114-bf7d-bda26556d116" (UID: "89adbd4d-061c-4114-bf7d-bda26556d116"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:02:02.344587 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.344555    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv" (OuterVolumeSpecName: "kube-api-access-nddzv") pod "89adbd4d-061c-4114-bf7d-bda26556d116" (UID: "89adbd4d-061c-4114-bf7d-bda26556d116"). InnerVolumeSpecName "kube-api-access-nddzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:02:02.344689 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.344585    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89adbd4d-061c-4114-bf7d-bda26556d116" (UID: "89adbd4d-061c-4114-bf7d-bda26556d116"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:02:02.344689 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.344593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm" (OuterVolumeSpecName: "kube-api-access-ng2cm") pod "c2e51ded-f193-480e-ad32-3fd353bac78b" (UID: "c2e51ded-f193-480e-ad32-3fd353bac78b"). InnerVolumeSpecName "kube-api-access-ng2cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:02:02.344763 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.344682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg" (OuterVolumeSpecName: "kube-api-access-rvpzg") pod "4804e0d9-975f-4039-801a-814dcf40fe3a" (UID: "4804e0d9-975f-4039-801a-814dcf40fe3a"). InnerVolumeSpecName "kube-api-access-rvpzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:02:02.428747 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.428653    2582 generic.go:356] "Generic (PLEG): container finished" podID="c2e51ded-f193-480e-ad32-3fd353bac78b" containerID="3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b" exitCode=0
Aug 04 23:02:02.428747 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.428700    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"
Aug 04 23:02:02.428980 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.428740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" event={"ID":"c2e51ded-f193-480e-ad32-3fd353bac78b","Type":"ContainerDied","Data":"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"}
Aug 04 23:02:02.428980 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.428797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2" event={"ID":"c2e51ded-f193-480e-ad32-3fd353bac78b","Type":"ContainerDied","Data":"101d9130a51b943cc4ce045475326f5d7fbd94208e1034fcdd99ed755bea043b"}
Aug 04 23:02:02.428980 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.428820    2582 scope.go:117] "RemoveContainer" containerID="3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"
Aug 04 23:02:02.430157 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.430134    2582 generic.go:356] "Generic (PLEG): container finished" podID="4804e0d9-975f-4039-801a-814dcf40fe3a" containerID="0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785" exitCode=0
Aug 04 23:02:02.430241 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.430187    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"
Aug 04 23:02:02.430241 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.430213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" event={"ID":"4804e0d9-975f-4039-801a-814dcf40fe3a","Type":"ContainerDied","Data":"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"}
Aug 04 23:02:02.430359 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.430264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn" event={"ID":"4804e0d9-975f-4039-801a-814dcf40fe3a","Type":"ContainerDied","Data":"8255a5a3ca7443a7ce7659a8bedbe432b74aa7a9f16cd9e147b169bedc8ab06e"}
Aug 04 23:02:02.431773 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.431750    2582 generic.go:356] "Generic (PLEG): container finished" podID="89adbd4d-061c-4114-bf7d-bda26556d116" containerID="a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64" exitCode=0
Aug 04 23:02:02.431971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.431799    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"
Aug 04 23:02:02.431971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.431815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" event={"ID":"89adbd4d-061c-4114-bf7d-bda26556d116","Type":"ContainerDied","Data":"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"}
Aug 04 23:02:02.431971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.431856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng" event={"ID":"89adbd4d-061c-4114-bf7d-bda26556d116","Type":"ContainerDied","Data":"e44d6c2eef823ca367cb0af6dd971558414deca1852c5004277bd78edc60cd62"}
Aug 04 23:02:02.439476 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.439457    2582 scope.go:117] "RemoveContainer" containerID="3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"
Aug 04 23:02:02.439750 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:02:02.439729    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b\": container with ID starting with 3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b not found: ID does not exist" containerID="3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"
Aug 04 23:02:02.439811 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.439755    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b"} err="failed to get container status \"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b\": rpc error: code = NotFound desc = could not find container \"3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b\": container with ID starting with 3dbf2928295596266b6d94c09137c7eec04804ebb2bcf10e0794f8c27e75ce2b not found: ID does not exist"
Aug 04 23:02:02.439811 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.439771    2582 scope.go:117] "RemoveContainer" containerID="0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"
Aug 04 23:02:02.443569 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443547    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2cm\" (UniqueName: \"kubernetes.io/projected/c2e51ded-f193-480e-ad32-3fd353bac78b-kube-api-access-ng2cm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443572    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4804e0d9-975f-4039-801a-814dcf40fe3a-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443583    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89adbd4d-061c-4114-bf7d-bda26556d116-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443591    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e51ded-f193-480e-ad32-3fd353bac78b-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443600    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddzv\" (UniqueName: \"kubernetes.io/projected/89adbd4d-061c-4114-bf7d-bda26556d116-kube-api-access-nddzv\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443608    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89adbd4d-061c-4114-bf7d-bda26556d116-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.443647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.443615    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpzg\" (UniqueName: \"kubernetes.io/projected/4804e0d9-975f-4039-801a-814dcf40fe3a-kube-api-access-rvpzg\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:02.449188 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.449171    2582 scope.go:117] "RemoveContainer" containerID="0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"
Aug 04 23:02:02.449410 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:02:02.449393    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785\": container with ID starting with 0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785 not found: ID does not exist" containerID="0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"
Aug 04 23:02:02.449461 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.449416    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785"} err="failed to get container status \"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785\": rpc error: code = NotFound desc = could not find container \"0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785\": container with ID starting with 0d3bd19fe27c710e278e666e6428983ff78f3b0578dc06d3cdf8358abc94d785 not found: ID does not exist"
Aug 04 23:02:02.449461 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.449436    2582 scope.go:117] "RemoveContainer" containerID="a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"
Aug 04 23:02:02.455757 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.455730    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:02:02.458318 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.458221    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e083a768-69cb4fc69b-m5gh2"]
Aug 04 23:02:02.458379 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.458333    2582 scope.go:117] "RemoveContainer" containerID="a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"
Aug 04 23:02:02.458596 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:02:02.458578    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64\": container with ID starting with a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64 not found: ID does not exist" containerID="a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"
Aug 04 23:02:02.458650 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.458603    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64"} err="failed to get container status \"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64\": rpc error: code = NotFound desc = could not find container \"a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64\": container with ID starting with a8c56596431deb80b450504d71c33cebd77d4e93321d14f0dd91888af6a72f64 not found: ID does not exist"
Aug 04 23:02:02.470616 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.470592    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:02:02.475638 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.475610    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e083a768-54fdfc5ccc-b4jng"]
Aug 04 23:02:02.486139 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.486116    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:02:02.489785 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:02.489765    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e083a768-66b8985754-v7xwn"]
Aug 04 23:02:03.018165 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:03.018137    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4804e0d9-975f-4039-801a-814dcf40fe3a" path="/var/lib/kubelet/pods/4804e0d9-975f-4039-801a-814dcf40fe3a/volumes"
Aug 04 23:02:03.018526 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:03.018512    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89adbd4d-061c-4114-bf7d-bda26556d116" path="/var/lib/kubelet/pods/89adbd4d-061c-4114-bf7d-bda26556d116/volumes"
Aug 04 23:02:03.018818 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:03.018807    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e51ded-f193-480e-ad32-3fd353bac78b" path="/var/lib/kubelet/pods/c2e51ded-f193-480e-ad32-3fd353bac78b/volumes"
Aug 04 23:02:05.281651 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:05.281610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:10.259006 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:10.258974    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:02:10.306585 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:10.306530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:25.279086 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.279042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" podUID="a73f91fc-83e3-44c4-94e7-8004219110be" containerName="main" containerID="cri-o://c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed" gracePeriod=45
Aug 04 23:02:25.534017 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.533960    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:02:25.542041 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.542013    2582 generic.go:356] "Generic (PLEG): container finished" podID="a73f91fc-83e3-44c4-94e7-8004219110be" containerID="c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed" exitCode=0
Aug 04 23:02:25.542157 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.542045    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"
Aug 04 23:02:25.542157 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.542098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" event={"ID":"a73f91fc-83e3-44c4-94e7-8004219110be","Type":"ContainerDied","Data":"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"}
Aug 04 23:02:25.542157 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.542133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk" event={"ID":"a73f91fc-83e3-44c4-94e7-8004219110be","Type":"ContainerDied","Data":"8cabe1cfd4e80da2015a12f2c26d153ca67cf9262b1d5fe93f2d5555186957eb"}
Aug 04 23:02:25.542157 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.542149    2582 scope.go:117] "RemoveContainer" containerID="c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"
Aug 04 23:02:25.551417 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.551363    2582 scope.go:117] "RemoveContainer" containerID="c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"
Aug 04 23:02:25.551795 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:02:25.551769    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed\": container with ID starting with c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed not found: ID does not exist" containerID="c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"
Aug 04 23:02:25.551955 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.551807    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed"} err="failed to get container status \"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed\": rpc error: code = NotFound desc = could not find container \"c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed\": container with ID starting with c1ae884fbc0b307d758ac6fa2a6338e576ed9339797bca8eda98a852a7083aed not found: ID does not exist"
Aug 04 23:02:25.639975 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.639947    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2pnq\" (UniqueName: \"kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640091    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640371 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640197    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:02:25.640371 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640240    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs\") pod \"a73f91fc-83e3-44c4-94e7-8004219110be\" (UID: \"a73f91fc-83e3-44c4-94e7-8004219110be\") "
Aug 04 23:02:25.640455 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache" (OuterVolumeSpecName: "model-cache") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:02:25.640455 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home" (OuterVolumeSpecName: "home") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:02:25.640521 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640501    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.640570 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640519    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.640570 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.640533    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.642360 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.642340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm" (OuterVolumeSpecName: "dshm") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:02:25.642423 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.642394    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:02:25.642459 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.642431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq" (OuterVolumeSpecName: "kube-api-access-j2pnq") pod "a73f91fc-83e3-44c4-94e7-8004219110be" (UID: "a73f91fc-83e3-44c4-94e7-8004219110be"). InnerVolumeSpecName "kube-api-access-j2pnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:02:25.741179 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.741153    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73f91fc-83e3-44c4-94e7-8004219110be-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.741179 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.741174    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73f91fc-83e3-44c4-94e7-8004219110be-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.741179 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.741184    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pnq\" (UniqueName: \"kubernetes.io/projected/a73f91fc-83e3-44c4-94e7-8004219110be-kube-api-access-j2pnq\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:02:25.864212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.864177    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:02:25.867842 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:25.867818    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e083a768/e2e-lifecycle-model-e083a768-kserve-787698d68c-pw5bk"]
Aug 04 23:02:27.011339 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:27.011297    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73f91fc-83e3-44c4-94e7-8004219110be" path="/var/lib/kubelet/pods/a73f91fc-83e3-44c4-94e7-8004219110be/volumes"
Aug 04 23:02:47.099089 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:47.099058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:47.517213 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:47.517173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:52.970523 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:52.970447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:53.025914 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:53.025865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:53.254931 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:53.254840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:02:53.482459 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:02:53.482414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:04.826042 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:04.826009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:04.921145 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:04.921115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:05.022183 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:05.022138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:05.621063 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:05.621021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:05.720542 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:05.720516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:05.820510 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:05.820480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:07.147019 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:07.146987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:14.366142 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:14.366105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:14.466045 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:14.466012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:15.074327 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:15.074272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:22.257726 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:22.257672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:22.366511 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:22.366466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:22.455111 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:22.455076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:33.016790 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:33.016743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:33.248303 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:33.248270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:46.870711 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:46.870667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:46.972537 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:46.972503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:03:54.201707 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:03:54.201670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:00.077202 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:00.077164    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:00.177260 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:00.177211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:00.507555 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:00.507511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:00.781936 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:00.781804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:01.013795 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:01.013761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:03.977265 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:03.977234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:04.076839 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:04.076798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:05.064057 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064026    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:04:05.064419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064405    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73f91fc-83e3-44c4-94e7-8004219110be" containerName="main"
Aug 04 23:04:05.064419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73f91fc-83e3-44c4-94e7-8004219110be" containerName="main"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064435    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e51ded-f193-480e-ad32-3fd353bac78b" containerName="payload-pre-processing"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064441    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e51ded-f193-480e-ad32-3fd353bac78b" containerName="payload-pre-processing"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064453    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4804e0d9-975f-4039-801a-814dcf40fe3a" containerName="payload-processing"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4804e0d9-975f-4039-801a-814dcf40fe3a" containerName="payload-processing"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064483    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89adbd4d-061c-4114-bf7d-bda26556d116" containerName="maas-api"
Aug 04 23:04:05.064533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064491    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89adbd4d-061c-4114-bf7d-bda26556d116" containerName="maas-api"
Aug 04 23:04:05.064725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064557    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4804e0d9-975f-4039-801a-814dcf40fe3a" containerName="payload-processing"
Aug 04 23:04:05.064725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064566    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73f91fc-83e3-44c4-94e7-8004219110be" containerName="main"
Aug 04 23:04:05.064725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064572    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89adbd4d-061c-4114-bf7d-bda26556d116" containerName="maas-api"
Aug 04 23:04:05.064725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.064578    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e51ded-f193-480e-ad32-3fd353bac78b" containerName="payload-pre-processing"
Aug 04 23:04:05.067997 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.067880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.070246 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.070224    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-549992-serving-cert\""
Aug 04 23:04:05.077266 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.077240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:04:05.099541 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.099519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:05.205025 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.205003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.205147 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.205121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.205192 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.205165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfhk\" (UniqueName: \"kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.305822 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.305796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.305954 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.305835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfhk\" (UniqueName: \"kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.305954 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.305857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.306049 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:04:05.306033    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-serving-cert: secret "maas-api-e2e-inf-a-549992-serving-cert" not found
Aug 04 23:04:05.306118 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:04:05.306103    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls podName:23fec744-daab-4a48-993a-f2d85f562d33 nodeName:}" failed. No retries permitted until 2026-08-04 23:04:05.806085458 +0000 UTC m=+2583.347578611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls") pod "maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" (UID: "23fec744-daab-4a48-993a-f2d85f562d33") : secret "maas-api-e2e-inf-a-549992-serving-cert" not found
Aug 04 23:04:05.306417 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.306399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.314382 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.314330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfhk\" (UniqueName: \"kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.679043 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.679011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:05.779629 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.779598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:05.790527 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.790500    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:04:05.793778 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.793753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:05.796485 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.796463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-549992-dockercfg-qdgp2\""
Aug 04 23:04:05.797120 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.797090    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-549992\""
Aug 04 23:04:05.804123 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.804099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:04:05.811522 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.811501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.813909 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.813878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:05.823933 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.823907    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:04:05.827263 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.827247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:05.838624 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.838598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:04:05.913087 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.913057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:05.913227 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.913105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55w6\" (UniqueName: \"kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:05.913227 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.913182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:05.913306 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.913256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqtv\" (UniqueName: \"kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:05.979792 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:05.979708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:06.014736 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.014697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqtv\" (UniqueName: \"kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:06.014889 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.014782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:06.014889 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.014801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55w6\" (UniqueName: \"kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:06.014889 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.014832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:06.015665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.015638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:06.015722 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.015690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:06.023634 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.023609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqtv\" (UniqueName: \"kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv\") pod \"payload-processing-e2e-inf-a-549992-598c564866-lrkhp\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:06.024448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.024421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55w6\" (UniqueName: \"kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6\") pod \"payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:06.107411 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.107379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:06.139008 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.138981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:06.256776 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.256683    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:04:06.261835 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:04:06.261792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5195ce1_425b_4e90_9ea7_04552e401e34.slice/crio-5e721606c203d6d25a9507807bb44adc14cd09e62761cfc4e453481430b03c96 WatchSource:0}: Error finding container 5e721606c203d6d25a9507807bb44adc14cd09e62761cfc4e453481430b03c96: Status 404 returned error can't find the container with id 5e721606c203d6d25a9507807bb44adc14cd09e62761cfc4e453481430b03c96
Aug 04 23:04:06.278187 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.278158    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:04:06.279756 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:04:06.279732    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b87a52_95f3_421a_8580_e2525bd338cd.slice/crio-5c95a1f9e379de2e48c42b4a99a07256c656033081fedd7c795c021ced52ca83 WatchSource:0}: Error finding container 5c95a1f9e379de2e48c42b4a99a07256c656033081fedd7c795c021ced52ca83: Status 404 returned error can't find the container with id 5c95a1f9e379de2e48c42b4a99a07256c656033081fedd7c795c021ced52ca83
Aug 04 23:04:06.329587 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:06.329509    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:04:06.330586 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:04:06.330558    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fec744_daab_4a48_993a_f2d85f562d33.slice/crio-9b890143e694f6bdd1e32e28501e4e85f72d0eb94bfcd2b1d2c54cd798c7c9fa WatchSource:0}: Error finding container 9b890143e694f6bdd1e32e28501e4e85f72d0eb94bfcd2b1d2c54cd798c7c9fa: Status 404 returned error can't find the container with id 9b890143e694f6bdd1e32e28501e4e85f72d0eb94bfcd2b1d2c54cd798c7c9fa
Aug 04 23:04:07.031568 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.031529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" event={"ID":"b5195ce1-425b-4e90-9ea7-04552e401e34","Type":"ContainerStarted","Data":"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"}
Aug 04 23:04:07.031757 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.031579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" event={"ID":"b5195ce1-425b-4e90-9ea7-04552e401e34","Type":"ContainerStarted","Data":"5e721606c203d6d25a9507807bb44adc14cd09e62761cfc4e453481430b03c96"}
Aug 04 23:04:07.031757 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.031619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:07.033272 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.033250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" event={"ID":"23fec744-daab-4a48-993a-f2d85f562d33","Type":"ContainerStarted","Data":"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"}
Aug 04 23:04:07.033371 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.033320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" event={"ID":"23fec744-daab-4a48-993a-f2d85f562d33","Type":"ContainerStarted","Data":"9b890143e694f6bdd1e32e28501e4e85f72d0eb94bfcd2b1d2c54cd798c7c9fa"}
Aug 04 23:04:07.033371 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.033342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:07.034742 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.034725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" event={"ID":"06b87a52-95f3-421a-8580-e2525bd338cd","Type":"ContainerStarted","Data":"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"}
Aug 04 23:04:07.034825 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.034758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" event={"ID":"06b87a52-95f3-421a-8580-e2525bd338cd","Type":"ContainerStarted","Data":"5c95a1f9e379de2e48c42b4a99a07256c656033081fedd7c795c021ced52ca83"}
Aug 04 23:04:07.034942 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.034924    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:07.051176 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.051092    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" podStartSLOduration=2.051078795 podStartE2EDuration="2.051078795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:04:07.050015693 +0000 UTC m=+2584.591508869" watchObservedRunningTime="2026-08-04 23:04:07.051078795 +0000 UTC m=+2584.592571969"
Aug 04 23:04:07.073762 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.073714    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" podStartSLOduration=1.6170383670000001 podStartE2EDuration="2.073696609s" totalImagesPullingTime="456.658242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:04:07.071737989 +0000 UTC m=+2584.613231165" watchObservedRunningTime="2026-08-04 23:04:07.073696609 +0000 UTC m=+2584.615189784"
Aug 04 23:04:07.117675 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:07.117630    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" podStartSLOduration=2.117617679 podStartE2EDuration="2.117617679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:04:07.114917228 +0000 UTC m=+2584.656410393" watchObservedRunningTime="2026-08-04 23:04:07.117617679 +0000 UTC m=+2584.659110881"
Aug 04 23:04:13.045766 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:13.045734    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:04:18.042095 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:18.042060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:04:18.042680 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:18.042145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:04:34.107454 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:34.107368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:34.433923 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:34.433858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:34.733287 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:34.733204    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:35.443273 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:35.443240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:37.548448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:37.548408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:37.645465 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:37.645438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:38.694446 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:38.694400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:38.950224 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:38.950128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:39.273435 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.273346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:39.372367 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.372330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:04:39.414443 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.414410    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:04:39.418162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.418139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.420432 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.420411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2d20db\""
Aug 04 23:04:39.420552 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.420432    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2d20db-dockercfg-n5vv8\""
Aug 04 23:04:39.430243 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.430219    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:04:39.458575 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.458550    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:04:39.462022 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.462006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.472856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.472830    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:04:39.515968 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.515940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.516108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.515992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjdk\" (UniqueName: \"kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.516108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.516017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.516226 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.516139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrht\" (UniqueName: \"kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.617390 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.617355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrht\" (UniqueName: \"kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.617532 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.617433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.617532 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.617486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjdk\" (UniqueName: \"kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.617532 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.617520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.618272 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.618250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.618370 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.618251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.625758 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.625736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjdk\" (UniqueName: \"kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk\") pod \"payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.626012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.625993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrht\" (UniqueName: \"kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht\") pod \"payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.730852 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.730830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:39.774053 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.774017    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:39.872249 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.872134    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:04:39.875145 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:04:39.875117    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b59a85_b3ef_4236_a1ec_455b6bde5111.slice/crio-dc4ed0f06c4758c092675a2c53bcfe3e4b089fa578b5e920700342d97d13d688 WatchSource:0}: Error finding container dc4ed0f06c4758c092675a2c53bcfe3e4b089fa578b5e920700342d97d13d688: Status 404 returned error can't find the container with id dc4ed0f06c4758c092675a2c53bcfe3e4b089fa578b5e920700342d97d13d688
Aug 04 23:04:39.929798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:39.929773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:04:39.931588 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:04:39.931563    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d51f23e_eec7_4c9c_81c4_70be39f75eb2.slice/crio-0d2d88b18162e4759ed99e7eb14d30c7ce439be948b733b8baa13fd4bd2009eb WatchSource:0}: Error finding container 0d2d88b18162e4759ed99e7eb14d30c7ce439be948b733b8baa13fd4bd2009eb: Status 404 returned error can't find the container with id 0d2d88b18162e4759ed99e7eb14d30c7ce439be948b733b8baa13fd4bd2009eb
Aug 04 23:04:40.198283 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.198193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" event={"ID":"6d51f23e-eec7-4c9c-81c4-70be39f75eb2","Type":"ContainerStarted","Data":"4220cb47c1b11cd8e402b2a9f0deb081f5f6bae9237d29ae5ae7252095d5f2ff"}
Aug 04 23:04:40.198283 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.198252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" event={"ID":"6d51f23e-eec7-4c9c-81c4-70be39f75eb2","Type":"ContainerStarted","Data":"0d2d88b18162e4759ed99e7eb14d30c7ce439be948b733b8baa13fd4bd2009eb"}
Aug 04 23:04:40.198713 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.198404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:04:40.200031 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.199999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" event={"ID":"e0b59a85-b3ef-4236-a1ec-455b6bde5111","Type":"ContainerStarted","Data":"edf4cd7d87de321b4b5fdc1f970a4341e3854ecdc66dc0f2521f71a186fd03b0"}
Aug 04 23:04:40.200163 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.200046    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:04:40.200163 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.200064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" event={"ID":"e0b59a85-b3ef-4236-a1ec-455b6bde5111","Type":"ContainerStarted","Data":"dc4ed0f06c4758c092675a2c53bcfe3e4b089fa578b5e920700342d97d13d688"}
Aug 04 23:04:40.217198 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.217142    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" podStartSLOduration=1.217129717 podStartE2EDuration="1.217129717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:04:40.214874827 +0000 UTC m=+2617.756368003" watchObservedRunningTime="2026-08-04 23:04:40.217129717 +0000 UTC m=+2617.758622890"
Aug 04 23:04:40.230711 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:40.230665    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" podStartSLOduration=1.23065282 podStartE2EDuration="1.23065282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:04:40.229185295 +0000 UTC m=+2617.770678470" watchObservedRunningTime="2026-08-04 23:04:40.23065282 +0000 UTC m=+2617.772146026"
Aug 04 23:04:51.206769 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:51.206726    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.62:9004: connect: connection refused"
Aug 04 23:04:51.207287 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:04:51.207014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:05:01.206317 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:01.206285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:05:01.994695 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:01.994660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:05:02.534291 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:02.534259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:05:10.822583 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:10.822545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:05:42.605006 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.604970    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:05:42.608456 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.608438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.610700 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.610680    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d20db\"/\"kube-root-ca.crt\""
Aug 04 23:05:42.611584 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.611558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d20db\"/\"default-dockercfg-zplrw\""
Aug 04 23:05:42.611697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.611558    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d20db\"/\"openshift-service-ca.crt\""
Aug 04 23:05:42.611697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.611652    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d20db\"/\"test-model-2d20db-kserve-self-signed-certs\""
Aug 04 23:05:42.615682 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.615663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:05:42.622957 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.622934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:05:42.767295 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.767295 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.767502 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.767502 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps47g\" (UniqueName: \"kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.767502 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.767502 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.767449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps47g\" (UniqueName: \"kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.868800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.868790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.869093 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.869034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.869171 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.869153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.869213 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.869193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.871023 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.871006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.871365 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.871348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.877129 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.877104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps47g\" (UniqueName: \"kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g\") pod \"test-model-2d20db-kserve-c9f69cdfc-5gncp\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") " pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:42.919151 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:42.919129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:43.059182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:43.059156    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:05:43.060853 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:05:43.060827    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c278998_dc7d_48df_b88a_c8e03e196b5f.slice/crio-72001479ed6cfc24bd0336704f489dd4393183ed2c6f464e6071b8a1ba15d252 WatchSource:0}: Error finding container 72001479ed6cfc24bd0336704f489dd4393183ed2c6f464e6071b8a1ba15d252: Status 404 returned error can't find the container with id 72001479ed6cfc24bd0336704f489dd4393183ed2c6f464e6071b8a1ba15d252
Aug 04 23:05:43.512186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:43.512143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" event={"ID":"2c278998-dc7d-48df-b88a-c8e03e196b5f","Type":"ContainerStarted","Data":"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"}
Aug 04 23:05:43.512186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:43.512190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" event={"ID":"2c278998-dc7d-48df-b88a-c8e03e196b5f","Type":"ContainerStarted","Data":"72001479ed6cfc24bd0336704f489dd4393183ed2c6f464e6071b8a1ba15d252"}
Aug 04 23:05:43.530103 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:43.530060    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" podStartSLOduration=1.530045167 podStartE2EDuration="1.530045167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:05:43.529272879 +0000 UTC m=+2681.070766053" watchObservedRunningTime="2026-08-04 23:05:43.530045167 +0000 UTC m=+2681.071538341"
Aug 04 23:05:52.919760 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:52.919724    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:52.920223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:52.919790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:52.932323 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:52.932300    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:05:53.573837 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:05:53.573798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:06:03.100757 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:03.100724    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:06:03.106439 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:03.106417    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:06:22.230338 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:22.230284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:23.275411 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:23.275376    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:06:23.296488 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:23.296464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:33.902592 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.902559    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:06:33.903165 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.902810    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" podUID="23fec744-daab-4a48-993a-f2d85f562d33" containerName="maas-api" containerID="cri-o://a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83" gracePeriod=30
Aug 04 23:06:33.933221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.933189    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:33.957629 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.957599    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:06:33.957872 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.957840    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" podUID="06b87a52-95f3-421a-8580-e2525bd338cd" containerName="payload-processing" containerID="cri-o://3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12" gracePeriod=30
Aug 04 23:06:33.965512 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.965482    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:06:33.965745 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.965702    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" podUID="b5195ce1-425b-4e90-9ea7-04552e401e34" containerName="payload-pre-processing" containerID="cri-o://567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67" gracePeriod=30
Aug 04 23:06:33.998540 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:33.998482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:34.054104 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.054071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:34.112557 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.112526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:34.269209 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.269178    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:06:34.272254 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.272233    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:06:34.358765 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.358743    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:06:34.430380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjqtv\" (UniqueName: \"kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv\") pod \"06b87a52-95f3-421a-8580-e2525bd338cd\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") "
Aug 04 23:06:34.430380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55w6\" (UniqueName: \"kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6\") pod \"b5195ce1-425b-4e90-9ea7-04552e401e34\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") "
Aug 04 23:06:34.430597 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430415    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume\") pod \"06b87a52-95f3-421a-8580-e2525bd338cd\" (UID: \"06b87a52-95f3-421a-8580-e2525bd338cd\") "
Aug 04 23:06:34.430597 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume\") pod \"b5195ce1-425b-4e90-9ea7-04552e401e34\" (UID: \"b5195ce1-425b-4e90-9ea7-04552e401e34\") "
Aug 04 23:06:34.430819 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430792    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06b87a52-95f3-421a-8580-e2525bd338cd" (UID: "06b87a52-95f3-421a-8580-e2525bd338cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:06:34.430886 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.430822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5195ce1-425b-4e90-9ea7-04552e401e34" (UID: "b5195ce1-425b-4e90-9ea7-04552e401e34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:06:34.432516 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.432497    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv" (OuterVolumeSpecName: "kube-api-access-gjqtv") pod "06b87a52-95f3-421a-8580-e2525bd338cd" (UID: "06b87a52-95f3-421a-8580-e2525bd338cd"). InnerVolumeSpecName "kube-api-access-gjqtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:34.432607 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.432584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6" (OuterVolumeSpecName: "kube-api-access-w55w6") pod "b5195ce1-425b-4e90-9ea7-04552e401e34" (UID: "b5195ce1-425b-4e90-9ea7-04552e401e34"). InnerVolumeSpecName "kube-api-access-w55w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:34.499818 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.499787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:34.532006 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.531969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktfhk\" (UniqueName: \"kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk\") pod \"23fec744-daab-4a48-993a-f2d85f562d33\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") "
Aug 04 23:06:34.532103 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532059    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") pod \"23fec744-daab-4a48-993a-f2d85f562d33\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") "
Aug 04 23:06:34.532160 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca\") pod \"23fec744-daab-4a48-993a-f2d85f562d33\" (UID: \"23fec744-daab-4a48-993a-f2d85f562d33\") "
Aug 04 23:06:34.532434 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532412    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23fec744-daab-4a48-993a-f2d85f562d33" (UID: "23fec744-daab-4a48-993a-f2d85f562d33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:06:34.532505 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532468    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqtv\" (UniqueName: \"kubernetes.io/projected/06b87a52-95f3-421a-8580-e2525bd338cd-kube-api-access-gjqtv\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.532505 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532486    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55w6\" (UniqueName: \"kubernetes.io/projected/b5195ce1-425b-4e90-9ea7-04552e401e34-kube-api-access-w55w6\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.532505 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532495    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b87a52-95f3-421a-8580-e2525bd338cd-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.532612 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.532508    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5195ce1-425b-4e90-9ea7-04552e401e34-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.534234 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.534213    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk" (OuterVolumeSpecName: "kube-api-access-ktfhk") pod "23fec744-daab-4a48-993a-f2d85f562d33" (UID: "23fec744-daab-4a48-993a-f2d85f562d33"). InnerVolumeSpecName "kube-api-access-ktfhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:34.534302 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.534249    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23fec744-daab-4a48-993a-f2d85f562d33" (UID: "23fec744-daab-4a48-993a-f2d85f562d33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:06:34.633717 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.633685    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfhk\" (UniqueName: \"kubernetes.io/projected/23fec744-daab-4a48-993a-f2d85f562d33-kube-api-access-ktfhk\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.633717 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.633713    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23fec744-daab-4a48-993a-f2d85f562d33-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.633867 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.633729    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fec744-daab-4a48-993a-f2d85f562d33-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:34.770836 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.770753    2582 generic.go:356] "Generic (PLEG): container finished" podID="b5195ce1-425b-4e90-9ea7-04552e401e34" containerID="567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67" exitCode=0
Aug 04 23:06:34.770836 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.770815    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"
Aug 04 23:06:34.771067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.770853    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" event={"ID":"b5195ce1-425b-4e90-9ea7-04552e401e34","Type":"ContainerDied","Data":"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"}
Aug 04 23:06:34.771067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.770981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl" event={"ID":"b5195ce1-425b-4e90-9ea7-04552e401e34","Type":"ContainerDied","Data":"5e721606c203d6d25a9507807bb44adc14cd09e62761cfc4e453481430b03c96"}
Aug 04 23:06:34.771067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.771038    2582 scope.go:117] "RemoveContainer" containerID="567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"
Aug 04 23:06:34.772339 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.772301    2582 generic.go:356] "Generic (PLEG): container finished" podID="23fec744-daab-4a48-993a-f2d85f562d33" containerID="a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83" exitCode=0
Aug 04 23:06:34.772459 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.772376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" event={"ID":"23fec744-daab-4a48-993a-f2d85f562d33","Type":"ContainerDied","Data":"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"}
Aug 04 23:06:34.772459 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.772407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj" event={"ID":"23fec744-daab-4a48-993a-f2d85f562d33","Type":"ContainerDied","Data":"9b890143e694f6bdd1e32e28501e4e85f72d0eb94bfcd2b1d2c54cd798c7c9fa"}
Aug 04 23:06:34.772568 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.772466    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"
Aug 04 23:06:34.774131 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.774091    2582 generic.go:356] "Generic (PLEG): container finished" podID="06b87a52-95f3-421a-8580-e2525bd338cd" containerID="3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12" exitCode=0
Aug 04 23:06:34.774257 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.774183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" event={"ID":"06b87a52-95f3-421a-8580-e2525bd338cd","Type":"ContainerDied","Data":"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"}
Aug 04 23:06:34.774257 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.774221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp" event={"ID":"06b87a52-95f3-421a-8580-e2525bd338cd","Type":"ContainerDied","Data":"5c95a1f9e379de2e48c42b4a99a07256c656033081fedd7c795c021ced52ca83"}
Aug 04 23:06:34.774379 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.774281    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"
Aug 04 23:06:34.782203 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.782188    2582 scope.go:117] "RemoveContainer" containerID="567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"
Aug 04 23:06:34.782525 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:06:34.782461    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67\": container with ID starting with 567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67 not found: ID does not exist" containerID="567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"
Aug 04 23:06:34.782631 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.782535    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67"} err="failed to get container status \"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67\": rpc error: code = NotFound desc = could not find container \"567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67\": container with ID starting with 567c55d2dea881bbc28e75e3c6a3053b0ade47420335f93b95100ba0443e1c67 not found: ID does not exist"
Aug 04 23:06:34.782631 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.782575    2582 scope.go:117] "RemoveContainer" containerID="a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"
Aug 04 23:06:34.791422 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.791406    2582 scope.go:117] "RemoveContainer" containerID="a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"
Aug 04 23:06:34.791653 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:06:34.791636    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83\": container with ID starting with a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83 not found: ID does not exist" containerID="a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"
Aug 04 23:06:34.791725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.791662    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83"} err="failed to get container status \"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83\": rpc error: code = NotFound desc = could not find container \"a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83\": container with ID starting with a10376f3109666b61b7fdf1f0c26c20aa753fc39c3fb4ec6932525d27c547e83 not found: ID does not exist"
Aug 04 23:06:34.791725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.791696    2582 scope.go:117] "RemoveContainer" containerID="3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"
Aug 04 23:06:34.800186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.800166    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:06:34.801162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.801142    2582 scope.go:117] "RemoveContainer" containerID="3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"
Aug 04 23:06:34.801462 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:06:34.801437    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12\": container with ID starting with 3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12 not found: ID does not exist" containerID="3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"
Aug 04 23:06:34.801544 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.801470    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12"} err="failed to get container status \"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12\": rpc error: code = NotFound desc = could not find container \"3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12\": container with ID starting with 3d0750955c299b039297fcec254a362a458be19fcae01e6dac23b3e985503f12 not found: ID does not exist"
Aug 04 23:06:34.802275 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.802255    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-549992-7c59cb4496-ggmzl"]
Aug 04 23:06:34.810246 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.810226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:06:34.812185 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.812166    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-549992-7d544f7c6f-rrqjj"]
Aug 04 23:06:34.821655 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.821632    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:06:34.827270 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:34.827249    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-549992-598c564866-lrkhp"]
Aug 04 23:06:35.011194 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:35.011168    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b87a52-95f3-421a-8580-e2525bd338cd" path="/var/lib/kubelet/pods/06b87a52-95f3-421a-8580-e2525bd338cd/volumes"
Aug 04 23:06:35.011605 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:35.011591    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23fec744-daab-4a48-993a-f2d85f562d33" path="/var/lib/kubelet/pods/23fec744-daab-4a48-993a-f2d85f562d33/volumes"
Aug 04 23:06:35.011961 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:35.011948    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5195ce1-425b-4e90-9ea7-04552e401e34" path="/var/lib/kubelet/pods/b5195ce1-425b-4e90-9ea7-04552e401e34/volumes"
Aug 04 23:06:38.295679 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.295636    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" podUID="2c278998-dc7d-48df-b88a-c8e03e196b5f" containerName="main" containerID="cri-o://8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8" gracePeriod=45
Aug 04 23:06:38.558099 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.558037    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:06:38.670555 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670521    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670593    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670665    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps47g\" (UniqueName: \"kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache\") pod \"2c278998-dc7d-48df-b88a-c8e03e196b5f\" (UID: \"2c278998-dc7d-48df-b88a-c8e03e196b5f\") "
Aug 04 23:06:38.670870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:06:38.671067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home" (OuterVolumeSpecName: "home") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:06:38.671067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.670985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:06:38.671067 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.671029    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.673053 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.673025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g" (OuterVolumeSpecName: "kube-api-access-ps47g") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "kube-api-access-ps47g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:38.673160 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.673052    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm" (OuterVolumeSpecName: "dshm") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:06:38.673160 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.673130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c278998-dc7d-48df-b88a-c8e03e196b5f" (UID: "2c278998-dc7d-48df-b88a-c8e03e196b5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:06:38.771834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.771799    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.771834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.771830    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.771834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.771839    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps47g\" (UniqueName: \"kubernetes.io/projected/2c278998-dc7d-48df-b88a-c8e03e196b5f-kube-api-access-ps47g\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.772094 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.771849    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c278998-dc7d-48df-b88a-c8e03e196b5f-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.772094 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.771859    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c278998-dc7d-48df-b88a-c8e03e196b5f-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:38.798086 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.798056    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c278998-dc7d-48df-b88a-c8e03e196b5f" containerID="8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8" exitCode=0
Aug 04 23:06:38.798221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.798101    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"
Aug 04 23:06:38.798221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.798136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" event={"ID":"2c278998-dc7d-48df-b88a-c8e03e196b5f","Type":"ContainerDied","Data":"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"}
Aug 04 23:06:38.798221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.798181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp" event={"ID":"2c278998-dc7d-48df-b88a-c8e03e196b5f","Type":"ContainerDied","Data":"72001479ed6cfc24bd0336704f489dd4393183ed2c6f464e6071b8a1ba15d252"}
Aug 04 23:06:38.798221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.798205    2582 scope.go:117] "RemoveContainer" containerID="8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"
Aug 04 23:06:38.806800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.806782    2582 scope.go:117] "RemoveContainer" containerID="8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"
Aug 04 23:06:38.807080 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:06:38.807051    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8\": container with ID starting with 8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8 not found: ID does not exist" containerID="8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"
Aug 04 23:06:38.807147 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.807082    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8"} err="failed to get container status \"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8\": rpc error: code = NotFound desc = could not find container \"8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8\": container with ID starting with 8b43b90625dda8e9b936f1e3e75792cbef5eff5606935310395a3e5d952d24c8 not found: ID does not exist"
Aug 04 23:06:38.818592 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.818541    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:06:38.822069 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:38.822048    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2d20db/test-model-2d20db-kserve-c9f69cdfc-5gncp"]
Aug 04 23:06:39.010741 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:39.010708    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c278998-dc7d-48df-b88a-c8e03e196b5f" path="/var/lib/kubelet/pods/2c278998-dc7d-48df-b88a-c8e03e196b5f/volumes"
Aug 04 23:06:40.568445 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:40.568403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:40.668148 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:40.668117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:40.829847 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:40.829767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:41.041752 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:41.041720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:51.620029 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.619990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:51.633196 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.633167    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:06:51.633413 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.633373    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" podUID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" containerName="payload-processing" containerID="cri-o://edf4cd7d87de321b4b5fdc1f970a4341e3854ecdc66dc0f2521f71a186fd03b0" gracePeriod=30
Aug 04 23:06:51.638613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.638583    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:06:51.640184 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.640137    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerName="payload-pre-processing" containerID="cri-o://4220cb47c1b11cd8e402b2a9f0deb081f5f6bae9237d29ae5ae7252095d5f2ff" gracePeriod=30
Aug 04 23:06:51.719247 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.719213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:51.814733 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.814703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:51.865112 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.865083    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerID="4220cb47c1b11cd8e402b2a9f0deb081f5f6bae9237d29ae5ae7252095d5f2ff" exitCode=0
Aug 04 23:06:51.865279 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.865158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" event={"ID":"6d51f23e-eec7-4c9c-81c4-70be39f75eb2","Type":"ContainerDied","Data":"4220cb47c1b11cd8e402b2a9f0deb081f5f6bae9237d29ae5ae7252095d5f2ff"}
Aug 04 23:06:51.866749 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.866722    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" containerID="edf4cd7d87de321b4b5fdc1f970a4341e3854ecdc66dc0f2521f71a186fd03b0" exitCode=0
Aug 04 23:06:51.866905 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.866872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" event={"ID":"e0b59a85-b3ef-4236-a1ec-455b6bde5111","Type":"ContainerDied","Data":"edf4cd7d87de321b4b5fdc1f970a4341e3854ecdc66dc0f2521f71a186fd03b0"}
Aug 04 23:06:51.917171 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.917150    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:06:51.930944 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.930923    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:06:51.985168 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lrht\" (UniqueName: \"kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht\") pod \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") "
Aug 04 23:06:51.985356 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfjdk\" (UniqueName: \"kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk\") pod \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") "
Aug 04 23:06:51.985356 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985245    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume\") pod \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\" (UID: \"6d51f23e-eec7-4c9c-81c4-70be39f75eb2\") "
Aug 04 23:06:51.985356 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985303    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume\") pod \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\" (UID: \"e0b59a85-b3ef-4236-a1ec-455b6bde5111\") "
Aug 04 23:06:51.985669 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985643    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d51f23e-eec7-4c9c-81c4-70be39f75eb2" (UID: "6d51f23e-eec7-4c9c-81c4-70be39f75eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:06:51.985763 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.985687    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0b59a85-b3ef-4236-a1ec-455b6bde5111" (UID: "e0b59a85-b3ef-4236-a1ec-455b6bde5111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:06:51.987520 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.987488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht" (OuterVolumeSpecName: "kube-api-access-5lrht") pod "6d51f23e-eec7-4c9c-81c4-70be39f75eb2" (UID: "6d51f23e-eec7-4c9c-81c4-70be39f75eb2"). InnerVolumeSpecName "kube-api-access-5lrht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:51.987520 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:51.987512    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk" (OuterVolumeSpecName: "kube-api-access-qfjdk") pod "e0b59a85-b3ef-4236-a1ec-455b6bde5111" (UID: "e0b59a85-b3ef-4236-a1ec-455b6bde5111"). InnerVolumeSpecName "kube-api-access-qfjdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:06:52.086141 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.086103    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjdk\" (UniqueName: \"kubernetes.io/projected/e0b59a85-b3ef-4236-a1ec-455b6bde5111-kube-api-access-qfjdk\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:52.086141 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.086134    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:52.086141 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.086146    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b59a85-b3ef-4236-a1ec-455b6bde5111-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:52.086380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.086158    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrht\" (UniqueName: \"kubernetes.io/projected/6d51f23e-eec7-4c9c-81c4-70be39f75eb2-kube-api-access-5lrht\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:06:52.217700 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.217618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:52.873592 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.873556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5" event={"ID":"6d51f23e-eec7-4c9c-81c4-70be39f75eb2","Type":"ContainerDied","Data":"0d2d88b18162e4759ed99e7eb14d30c7ce439be948b733b8baa13fd4bd2009eb"}
Aug 04 23:06:52.874075 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.873569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"
Aug 04 23:06:52.874075 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.873646    2582 scope.go:117] "RemoveContainer" containerID="4220cb47c1b11cd8e402b2a9f0deb081f5f6bae9237d29ae5ae7252095d5f2ff"
Aug 04 23:06:52.875116 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.875099    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"
Aug 04 23:06:52.875203 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.875135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk" event={"ID":"e0b59a85-b3ef-4236-a1ec-455b6bde5111","Type":"ContainerDied","Data":"dc4ed0f06c4758c092675a2c53bcfe3e4b089fa578b5e920700342d97d13d688"}
Aug 04 23:06:52.882787 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.882769    2582 scope.go:117] "RemoveContainer" containerID="edf4cd7d87de321b4b5fdc1f970a4341e3854ecdc66dc0f2521f71a186fd03b0"
Aug 04 23:06:52.898783 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.898762    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:06:52.900572 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.900552    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d20db-6789c77664-c98kk"]
Aug 04 23:06:52.910325 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.910274    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:06:52.913872 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:52.913851    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d20db-6b848965d4-h5bf5"]
Aug 04 23:06:53.010705 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:53.010675    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" path="/var/lib/kubelet/pods/6d51f23e-eec7-4c9c-81c4-70be39f75eb2/volumes"
Aug 04 23:06:53.011053 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:53.011040    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" path="/var/lib/kubelet/pods/e0b59a85-b3ef-4236-a1ec-455b6bde5111/volumes"
Aug 04 23:06:58.092791 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:58.092752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:58.194726 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:58.194694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:58.339488 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:58.339458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:58.556263 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:58.556231    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:58.926678 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:58.926650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:59.239407 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:59.239316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:06:59.372740 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:06:59.372711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:02.201338 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:02.201289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:02.300426 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:02.300377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:03.301066 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301029    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:07:03.301438 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301424    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23fec744-daab-4a48-993a-f2d85f562d33" containerName="maas-api"
Aug 04 23:07:03.301480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301441    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="23fec744-daab-4a48-993a-f2d85f562d33" containerName="maas-api"
Aug 04 23:07:03.301480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301460    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b87a52-95f3-421a-8580-e2525bd338cd" containerName="payload-processing"
Aug 04 23:07:03.301480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301466    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b87a52-95f3-421a-8580-e2525bd338cd" containerName="payload-processing"
Aug 04 23:07:03.301480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301475    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5195ce1-425b-4e90-9ea7-04552e401e34" containerName="payload-pre-processing"
Aug 04 23:07:03.301480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301481    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5195ce1-425b-4e90-9ea7-04552e401e34" containerName="payload-pre-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301490    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c278998-dc7d-48df-b88a-c8e03e196b5f" containerName="main"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c278998-dc7d-48df-b88a-c8e03e196b5f" containerName="main"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301508    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerName="payload-pre-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301513    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerName="payload-pre-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301529    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" containerName="payload-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301537    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" containerName="payload-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301591    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b59a85-b3ef-4236-a1ec-455b6bde5111" containerName="payload-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301602    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b87a52-95f3-421a-8580-e2525bd338cd" containerName="payload-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301613    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51f23e-eec7-4c9c-81c4-70be39f75eb2" containerName="payload-pre-processing"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301621    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="23fec744-daab-4a48-993a-f2d85f562d33" containerName="maas-api"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301627    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c278998-dc7d-48df-b88a-c8e03e196b5f" containerName="main"
Aug 04 23:07:03.301636 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.301635    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5195ce1-425b-4e90-9ea7-04552e401e34" containerName="payload-pre-processing"
Aug 04 23:07:03.306203 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.306187    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.308190 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.308169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8f0bc6-serving-cert\""
Aug 04 23:07:03.312596 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.312573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:07:03.390761 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.390717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.390971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.390841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndn5v\" (UniqueName: \"kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.390971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.390925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.400653 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.400624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:03.491884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.491854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndn5v\" (UniqueName: \"kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.492049 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.491890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.492049 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.491970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.492141 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:07:03.492093    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-serving-cert: secret "maas-api-e2e-api-a-8f0bc6-serving-cert" not found
Aug 04 23:07:03.492190 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:07:03.492179    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls podName:45250f29-2416-431e-826a-76eba783c54c nodeName:}" failed. No retries permitted until 2026-08-04 23:07:03.992163191 +0000 UTC m=+2761.533656343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls") pod "maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" (UID: "45250f29-2416-431e-826a-76eba783c54c") : secret "maas-api-e2e-api-a-8f0bc6-serving-cert" not found
Aug 04 23:07:03.492543 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.492527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.500766 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.500743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndn5v\" (UniqueName: \"kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.859431 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.859395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:03.956817 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.956780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:03.957364 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.957347    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:07:03.961023 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.961008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:03.963085 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.963065    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8f0bc6\""
Aug 04 23:07:03.969530 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.969477    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:07:03.998915 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.992290    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:07:03.998915 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.996484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:03.998915 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:07:03.996682    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-serving-cert: secret "maas-api-e2e-api-a-8f0bc6-serving-cert" not found
Aug 04 23:07:03.998915 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:07:03.996741    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls podName:45250f29-2416-431e-826a-76eba783c54c nodeName:}" failed. No retries permitted until 2026-08-04 23:07:04.996721411 +0000 UTC m=+2762.538214568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls") pod "maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" (UID: "45250f29-2416-431e-826a-76eba783c54c") : secret "maas-api-e2e-api-a-8f0bc6-serving-cert" not found
Aug 04 23:07:03.998915 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:03.997635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.006791 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.003785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8f0bc6-dockercfg-947qm\""
Aug 04 23:07:04.007394 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.007374    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:07:04.075822 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.075770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:04.098019 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.097990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.098135 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.098023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.098135 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.098116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbv25\" (UniqueName: \"kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.098214 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.098149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf62\" (UniqueName: \"kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.199015 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.198982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.199181 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.199025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.199181 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.199097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv25\" (UniqueName: \"kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.199181 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.199122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf62\" (UniqueName: \"kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.199674 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.199653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.199775 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.199755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.207026 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.207006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf62\" (UniqueName: \"kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62\") pod \"payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.207154 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.207133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv25\" (UniqueName: \"kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25\") pod \"payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.273293 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.273266    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.319686 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.319649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.431239 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.431209    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:07:04.433125 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:07:04.433094    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffa8c54_ba26_40d6_83cd_7c4068424159.slice/crio-810a0d4425590a0cb2c93e73b14e413ababce1574a0448a80eb0cbf97f9af509 WatchSource:0}: Error finding container 810a0d4425590a0cb2c93e73b14e413ababce1574a0448a80eb0cbf97f9af509: Status 404 returned error can't find the container with id 810a0d4425590a0cb2c93e73b14e413ababce1574a0448a80eb0cbf97f9af509
Aug 04 23:07:04.477687 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.477662    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:07:04.482949 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:07:04.482924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2551177d_0723_4a40_b39a_65ed6bc041bb.slice/crio-684bc4abd79c54151e1f7692e94cd903381cc04f87b990f3ef5448198eec15b7 WatchSource:0}: Error finding container 684bc4abd79c54151e1f7692e94cd903381cc04f87b990f3ef5448198eec15b7: Status 404 returned error can't find the container with id 684bc4abd79c54151e1f7692e94cd903381cc04f87b990f3ef5448198eec15b7
Aug 04 23:07:04.941303 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.941273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" event={"ID":"effa8c54-ba26-40d6-83cd-7c4068424159","Type":"ContainerStarted","Data":"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"}
Aug 04 23:07:04.941491 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.941330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" event={"ID":"effa8c54-ba26-40d6-83cd-7c4068424159","Type":"ContainerStarted","Data":"810a0d4425590a0cb2c93e73b14e413ababce1574a0448a80eb0cbf97f9af509"}
Aug 04 23:07:04.941491 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.941358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:04.942869 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.942844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" event={"ID":"2551177d-0723-4a40-b39a-65ed6bc041bb","Type":"ContainerStarted","Data":"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"}
Aug 04 23:07:04.943010 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.942885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" event={"ID":"2551177d-0723-4a40-b39a-65ed6bc041bb","Type":"ContainerStarted","Data":"684bc4abd79c54151e1f7692e94cd903381cc04f87b990f3ef5448198eec15b7"}
Aug 04 23:07:04.943010 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.942922    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:04.957075 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.957034    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" podStartSLOduration=1.9570199640000001 podStartE2EDuration="1.957019964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:07:04.955699313 +0000 UTC m=+2762.497192486" watchObservedRunningTime="2026-08-04 23:07:04.957019964 +0000 UTC m=+2762.498513138"
Aug 04 23:07:04.970183 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:04.970135    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" podStartSLOduration=1.9701167229999998 podStartE2EDuration="1.970116723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:07:04.969066206 +0000 UTC m=+2762.510559382" watchObservedRunningTime="2026-08-04 23:07:04.970116723 +0000 UTC m=+2762.511609898"
Aug 04 23:07:05.009393 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.008917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:05.013108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.013086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:05.119091 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.119066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:05.253419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.253397    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:07:05.255766 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:07:05.255737    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45250f29_2416_431e_826a_76eba783c54c.slice/crio-c5084f6df5099f8e7103e7f050fef5ae7d98134e6e9dfcee32c017705829de69 WatchSource:0}: Error finding container c5084f6df5099f8e7103e7f050fef5ae7d98134e6e9dfcee32c017705829de69: Status 404 returned error can't find the container with id c5084f6df5099f8e7103e7f050fef5ae7d98134e6e9dfcee32c017705829de69
Aug 04 23:07:05.257242 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.257223    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 23:07:05.950583 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.950544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" event={"ID":"45250f29-2416-431e-826a-76eba783c54c","Type":"ContainerStarted","Data":"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"}
Aug 04 23:07:05.951043 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.950595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" event={"ID":"45250f29-2416-431e-826a-76eba783c54c","Type":"ContainerStarted","Data":"c5084f6df5099f8e7103e7f050fef5ae7d98134e6e9dfcee32c017705829de69"}
Aug 04 23:07:05.951126 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.951107    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:05.973971 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:05.973865    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" podStartSLOduration=2.49342609 podStartE2EDuration="2.973844551s" totalImagesPullingTime="480.418461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:07:05.973008125 +0000 UTC m=+2763.514501324" watchObservedRunningTime="2026-08-04 23:07:05.973844551 +0000 UTC m=+2763.515337726"
Aug 04 23:07:12.967334 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:12.967296    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:07:15.952660 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:15.952627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:07:15.953415 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:15.953392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:07:27.103266 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:27.103185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:27.574770 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:27.574734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:27.896471 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:27.896432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:28.052984 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:28.052946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:31.252192 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:31.252157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:31.339958 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:31.339921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:32.383634 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:32.383596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:32.664638 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:32.664559    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:32.884191 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:32.884153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:32.979420 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:32.979327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:07:33.063504 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.063472    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:07:33.067232 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.067216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.070583 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.070562    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-25c344-dockercfg-x86vm\""
Aug 04 23:07:33.071586 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.071567    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-25c344\""
Aug 04 23:07:33.096617 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.096595    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:07:33.254875 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.254791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.254875 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.254828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxjb\" (UniqueName: \"kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.356128 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.356089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxjb\" (UniqueName: \"kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.356322 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.356247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.356948 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.356887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.364065 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.364046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxjb\" (UniqueName: \"kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb\") pod \"payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.377129 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.377106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:33.512293 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:33.512269    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:07:33.514940 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:07:33.514886    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc93611_52f2_4225_891e_7e6b4660c784.slice/crio-74302ba48907ca4cf5a9079e9e77a1b49763d566a20295cc8a18f90665c13fb4 WatchSource:0}: Error finding container 74302ba48907ca4cf5a9079e9e77a1b49763d566a20295cc8a18f90665c13fb4: Status 404 returned error can't find the container with id 74302ba48907ca4cf5a9079e9e77a1b49763d566a20295cc8a18f90665c13fb4
Aug 04 23:07:34.090127 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:34.090095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" event={"ID":"efc93611-52f2-4225-891e-7e6b4660c784","Type":"ContainerStarted","Data":"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"}
Aug 04 23:07:34.090312 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:34.090144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" event={"ID":"efc93611-52f2-4225-891e-7e6b4660c784","Type":"ContainerStarted","Data":"74302ba48907ca4cf5a9079e9e77a1b49763d566a20295cc8a18f90665c13fb4"}
Aug 04 23:07:34.090312 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:34.090232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:34.104227 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:34.104166    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" podStartSLOduration=1.104149095 podStartE2EDuration="1.104149095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:07:34.104135345 +0000 UTC m=+2791.645628521" watchObservedRunningTime="2026-08-04 23:07:34.104149095 +0000 UTC m=+2791.645642271"
Aug 04 23:07:45.097586 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:45.097550    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:07:56.231787 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:07:56.231752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:14.910823 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.910784    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:08:14.911312 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.911056    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" podUID="45250f29-2416-431e-826a-76eba783c54c" containerName="maas-api" containerID="cri-o://6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e" gracePeriod=30
Aug 04 23:08:14.942286 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.942258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:14.967401 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.967362    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:08:14.967620 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.967596    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" podUID="2551177d-0723-4a40-b39a-65ed6bc041bb" containerName="payload-processing" containerID="cri-o://8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2" gracePeriod=30
Aug 04 23:08:14.988695 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.980441    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:08:14.988695 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:14.980748    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" podUID="effa8c54-ba26-40d6-83cd-7c4068424159" containerName="payload-pre-processing" containerID="cri-o://ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409" gracePeriod=30
Aug 04 23:08:15.006109 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.006077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:15.110439 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.110393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:15.170977 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.170785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:15.241084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.241055    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:08:15.265272 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.265249    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:08:15.268401 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.268384    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:08:15.302136 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.302112    2582 generic.go:356] "Generic (PLEG): container finished" podID="2551177d-0723-4a40-b39a-65ed6bc041bb" containerID="8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2" exitCode=0
Aug 04 23:08:15.302275 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.302150    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"
Aug 04 23:08:15.302275 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.302195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" event={"ID":"2551177d-0723-4a40-b39a-65ed6bc041bb","Type":"ContainerDied","Data":"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"}
Aug 04 23:08:15.302275 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.302253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q" event={"ID":"2551177d-0723-4a40-b39a-65ed6bc041bb","Type":"ContainerDied","Data":"684bc4abd79c54151e1f7692e94cd903381cc04f87b990f3ef5448198eec15b7"}
Aug 04 23:08:15.302427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.302289    2582 scope.go:117] "RemoveContainer" containerID="8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"
Aug 04 23:08:15.303618 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.303597    2582 generic.go:356] "Generic (PLEG): container finished" podID="effa8c54-ba26-40d6-83cd-7c4068424159" containerID="ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409" exitCode=0
Aug 04 23:08:15.303728 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.303635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"
Aug 04 23:08:15.303728 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.303632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" event={"ID":"effa8c54-ba26-40d6-83cd-7c4068424159","Type":"ContainerDied","Data":"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"}
Aug 04 23:08:15.303848 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.303754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm" event={"ID":"effa8c54-ba26-40d6-83cd-7c4068424159","Type":"ContainerDied","Data":"810a0d4425590a0cb2c93e73b14e413ababce1574a0448a80eb0cbf97f9af509"}
Aug 04 23:08:15.305155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.305134    2582 generic.go:356] "Generic (PLEG): container finished" podID="45250f29-2416-431e-826a-76eba783c54c" containerID="6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e" exitCode=0
Aug 04 23:08:15.305278 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.305162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" event={"ID":"45250f29-2416-431e-826a-76eba783c54c","Type":"ContainerDied","Data":"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"}
Aug 04 23:08:15.305278 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.305174    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"
Aug 04 23:08:15.305278 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.305192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5" event={"ID":"45250f29-2416-431e-826a-76eba783c54c","Type":"ContainerDied","Data":"c5084f6df5099f8e7103e7f050fef5ae7d98134e6e9dfcee32c017705829de69"}
Aug 04 23:08:15.315630 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.315605    2582 scope.go:117] "RemoveContainer" containerID="8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"
Aug 04 23:08:15.315974 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:08:15.315951    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2\": container with ID starting with 8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2 not found: ID does not exist" containerID="8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"
Aug 04 23:08:15.316045 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.315985    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2"} err="failed to get container status \"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2\": rpc error: code = NotFound desc = could not find container \"8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2\": container with ID starting with 8d3955d3294ed7daa1553920983432342bd8d6f97619dac852c029f50368fab2 not found: ID does not exist"
Aug 04 23:08:15.316045 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.316004    2582 scope.go:117] "RemoveContainer" containerID="ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"
Aug 04 23:08:15.324115 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.324098    2582 scope.go:117] "RemoveContainer" containerID="ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"
Aug 04 23:08:15.324363 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:08:15.324347    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409\": container with ID starting with ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409 not found: ID does not exist" containerID="ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"
Aug 04 23:08:15.324431 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.324368    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409"} err="failed to get container status \"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409\": rpc error: code = NotFound desc = could not find container \"ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409\": container with ID starting with ba1c2c656f785f5e43d8959efffad3247b80817018c93253c1c0cc5e95338409 not found: ID does not exist"
Aug 04 23:08:15.324431 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.324382    2582 scope.go:117] "RemoveContainer" containerID="6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"
Aug 04 23:08:15.332095 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.332077    2582 scope.go:117] "RemoveContainer" containerID="6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"
Aug 04 23:08:15.332335 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:08:15.332317    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e\": container with ID starting with 6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e not found: ID does not exist" containerID="6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"
Aug 04 23:08:15.332395 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.332341    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e"} err="failed to get container status \"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e\": rpc error: code = NotFound desc = could not find container \"6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e\": container with ID starting with 6f968d393297239a633617469d3929a3aae8415112be5933e87b4fcf1b876d4e not found: ID does not exist"
Aug 04 23:08:15.406094 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndn5v\" (UniqueName: \"kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v\") pod \"45250f29-2416-431e-826a-76eba783c54c\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") "
Aug 04 23:08:15.406174 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume\") pod \"2551177d-0723-4a40-b39a-65ed6bc041bb\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") "
Aug 04 23:08:15.406218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca\") pod \"45250f29-2416-431e-826a-76eba783c54c\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") "
Aug 04 23:08:15.406218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406209    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbv25\" (UniqueName: \"kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25\") pod \"effa8c54-ba26-40d6-83cd-7c4068424159\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") "
Aug 04 23:08:15.406315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") pod \"45250f29-2416-431e-826a-76eba783c54c\" (UID: \"45250f29-2416-431e-826a-76eba783c54c\") "
Aug 04 23:08:15.406315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406295    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slf62\" (UniqueName: \"kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62\") pod \"2551177d-0723-4a40-b39a-65ed6bc041bb\" (UID: \"2551177d-0723-4a40-b39a-65ed6bc041bb\") "
Aug 04 23:08:15.406419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406344    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume\") pod \"effa8c54-ba26-40d6-83cd-7c4068424159\" (UID: \"effa8c54-ba26-40d6-83cd-7c4068424159\") "
Aug 04 23:08:15.406571 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2551177d-0723-4a40-b39a-65ed6bc041bb" (UID: "2551177d-0723-4a40-b39a-65ed6bc041bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:08:15.406571 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406540    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45250f29-2416-431e-826a-76eba783c54c" (UID: "45250f29-2416-431e-826a-76eba783c54c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:08:15.406697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406667    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2551177d-0723-4a40-b39a-65ed6bc041bb-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.406697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406686    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45250f29-2416-431e-826a-76eba783c54c-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.406937 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.406913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "effa8c54-ba26-40d6-83cd-7c4068424159" (UID: "effa8c54-ba26-40d6-83cd-7c4068424159"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:08:15.408458 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.408430    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25" (OuterVolumeSpecName: "kube-api-access-qbv25") pod "effa8c54-ba26-40d6-83cd-7c4068424159" (UID: "effa8c54-ba26-40d6-83cd-7c4068424159"). InnerVolumeSpecName "kube-api-access-qbv25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:08:15.408607 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.408588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62" (OuterVolumeSpecName: "kube-api-access-slf62") pod "2551177d-0723-4a40-b39a-65ed6bc041bb" (UID: "2551177d-0723-4a40-b39a-65ed6bc041bb"). InnerVolumeSpecName "kube-api-access-slf62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:08:15.408884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.408869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v" (OuterVolumeSpecName: "kube-api-access-ndn5v") pod "45250f29-2416-431e-826a-76eba783c54c" (UID: "45250f29-2416-431e-826a-76eba783c54c"). InnerVolumeSpecName "kube-api-access-ndn5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:08:15.408884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.408871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45250f29-2416-431e-826a-76eba783c54c" (UID: "45250f29-2416-431e-826a-76eba783c54c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:08:15.498619 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.498557    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:15.507755 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.507732    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbv25\" (UniqueName: \"kubernetes.io/projected/effa8c54-ba26-40d6-83cd-7c4068424159-kube-api-access-qbv25\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.507833 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.507756    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45250f29-2416-431e-826a-76eba783c54c-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.507833 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.507765    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slf62\" (UniqueName: \"kubernetes.io/projected/2551177d-0723-4a40-b39a-65ed6bc041bb-kube-api-access-slf62\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.507833 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.507774    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effa8c54-ba26-40d6-83cd-7c4068424159-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.507833 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.507782    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndn5v\" (UniqueName: \"kubernetes.io/projected/45250f29-2416-431e-826a-76eba783c54c-kube-api-access-ndn5v\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:15.624018 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.623997    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:08:15.627333 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.627311    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8f0bc6-9d458cf54-4tk7q"]
Aug 04 23:08:15.635975 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.635953    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:08:15.639933 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.639913    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8f0bc6-59b877784-68shm"]
Aug 04 23:08:15.649527 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.649508    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:08:15.652126 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:15.652107    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f0bc6-6b55448449-zrpz5"]
Aug 04 23:08:17.010831 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:17.010799    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2551177d-0723-4a40-b39a-65ed6bc041bb" path="/var/lib/kubelet/pods/2551177d-0723-4a40-b39a-65ed6bc041bb/volumes"
Aug 04 23:08:17.011227 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:17.011182    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45250f29-2416-431e-826a-76eba783c54c" path="/var/lib/kubelet/pods/45250f29-2416-431e-826a-76eba783c54c/volumes"
Aug 04 23:08:17.011499 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:17.011485    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="effa8c54-ba26-40d6-83cd-7c4068424159" path="/var/lib/kubelet/pods/effa8c54-ba26-40d6-83cd-7c4068424159/volumes"
Aug 04 23:08:21.190497 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:21.190469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:21.292471 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:21.292443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:21.468870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:21.468795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:21.739539 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:21.739460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:37.405022 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.404982    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:37.441735 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.441702    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:08:37.442736 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.442550    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" podUID="efc93611-52f2-4225-891e-7e6b4660c784" containerName="payload-pre-processing" containerID="cri-o://f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4" gracePeriod=30
Aug 04 23:08:37.472601 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.472556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:37.573579 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.573539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:37.629421 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.629393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:37.696691 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.696669    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:08:37.793420 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.793398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxjb\" (UniqueName: \"kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb\") pod \"efc93611-52f2-4225-891e-7e6b4660c784\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") "
Aug 04 23:08:37.793562 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.793494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume\") pod \"efc93611-52f2-4225-891e-7e6b4660c784\" (UID: \"efc93611-52f2-4225-891e-7e6b4660c784\") "
Aug 04 23:08:37.793861 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.793838    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efc93611-52f2-4225-891e-7e6b4660c784" (UID: "efc93611-52f2-4225-891e-7e6b4660c784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:08:37.795547 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.795521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb" (OuterVolumeSpecName: "kube-api-access-6hxjb") pod "efc93611-52f2-4225-891e-7e6b4660c784" (UID: "efc93611-52f2-4225-891e-7e6b4660c784"). InnerVolumeSpecName "kube-api-access-6hxjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:08:37.894734 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.894709    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxjb\" (UniqueName: \"kubernetes.io/projected/efc93611-52f2-4225-891e-7e6b4660c784-kube-api-access-6hxjb\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:37.894734 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.894733    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc93611-52f2-4225-891e-7e6b4660c784-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:08:37.956299 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:37.956224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:38.430934 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.430872    2582 generic.go:356] "Generic (PLEG): container finished" podID="efc93611-52f2-4225-891e-7e6b4660c784" containerID="f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4" exitCode=0
Aug 04 23:08:38.431354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.430936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"
Aug 04 23:08:38.431354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.430969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" event={"ID":"efc93611-52f2-4225-891e-7e6b4660c784","Type":"ContainerDied","Data":"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"}
Aug 04 23:08:38.431354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.431029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m" event={"ID":"efc93611-52f2-4225-891e-7e6b4660c784","Type":"ContainerDied","Data":"74302ba48907ca4cf5a9079e9e77a1b49763d566a20295cc8a18f90665c13fb4"}
Aug 04 23:08:38.431354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.431058    2582 scope.go:117] "RemoveContainer" containerID="f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"
Aug 04 23:08:38.440220 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.440199    2582 scope.go:117] "RemoveContainer" containerID="f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"
Aug 04 23:08:38.440462 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:08:38.440442    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4\": container with ID starting with f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4 not found: ID does not exist" containerID="f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"
Aug 04 23:08:38.440525 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.440471    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4"} err="failed to get container status \"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4\": rpc error: code = NotFound desc = could not find container \"f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4\": container with ID starting with f5aef72bc64ef28075ffb1dc81440dc0eaf44cf9eb4dac5ee5923c7a35e14ec4 not found: ID does not exist"
Aug 04 23:08:38.452504 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.452482    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:08:38.461774 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:38.461753    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25c344-567f49dcdd-nwk7m"]
Aug 04 23:08:39.010480 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:39.010450    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc93611-52f2-4225-891e-7e6b4660c784" path="/var/lib/kubelet/pods/efc93611-52f2-4225-891e-7e6b4660c784/volumes"
Aug 04 23:08:43.761171 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:43.761131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:43.823740 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:43.823705    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:44.034885 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:44.034802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:44.272626 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:44.272584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:44.619489 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:44.619458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:44.958890 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:44.958812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:45.745304 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:45.745258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:47.813436 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:47.813344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:47.912460 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:47.912430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:48.906814 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.906783    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907174    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effa8c54-ba26-40d6-83cd-7c4068424159" containerName="payload-pre-processing"
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907186    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="effa8c54-ba26-40d6-83cd-7c4068424159" containerName="payload-pre-processing"
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907195    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2551177d-0723-4a40-b39a-65ed6bc041bb" containerName="payload-processing"
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907201    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2551177d-0723-4a40-b39a-65ed6bc041bb" containerName="payload-processing"
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907208    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45250f29-2416-431e-826a-76eba783c54c" containerName="maas-api"
Aug 04 23:08:48.907212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907214    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="45250f29-2416-431e-826a-76eba783c54c" containerName="maas-api"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907220    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc93611-52f2-4225-891e-7e6b4660c784" containerName="payload-pre-processing"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907226    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc93611-52f2-4225-891e-7e6b4660c784" containerName="payload-pre-processing"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907295    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="45250f29-2416-431e-826a-76eba783c54c" containerName="maas-api"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907301    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc93611-52f2-4225-891e-7e6b4660c784" containerName="payload-pre-processing"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907309    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="effa8c54-ba26-40d6-83cd-7c4068424159" containerName="payload-pre-processing"
Aug 04 23:08:48.907608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.907314    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2551177d-0723-4a40-b39a-65ed6bc041bb" containerName="payload-processing"
Aug 04 23:08:48.910327 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.910309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:48.912488 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.912464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ec06a8-serving-cert\""
Aug 04 23:08:48.919730 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.919669    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:08:48.939675 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.939645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:48.996257 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.996232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmnm\" (UniqueName: \"kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:48.996376 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.996271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:48.996376 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:48.996355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.029687 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.029651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:49.097401 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.097380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmnm\" (UniqueName: \"kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.097506 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.097415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.097506 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.097459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.098065 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.098037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.099961 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.099940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.104993 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.104974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmnm\" (UniqueName: \"kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm\") pod \"maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.223474 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.223416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:49.373819 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.373794    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:08:49.376581 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:08:49.376545    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f89345_6a74_43a6_97e1_24854a4b3f25.slice/crio-2cb92766c4973243d2febf422333b4c73b548eb1c74fb2c5ccaf3da13e75a027 WatchSource:0}: Error finding container 2cb92766c4973243d2febf422333b4c73b548eb1c74fb2c5ccaf3da13e75a027: Status 404 returned error can't find the container with id 2cb92766c4973243d2febf422333b4c73b548eb1c74fb2c5ccaf3da13e75a027
Aug 04 23:08:49.429570 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.429539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:49.487154 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.487090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" event={"ID":"f4f89345-6a74-43a6-97e1-24854a4b3f25","Type":"ContainerStarted","Data":"2cb92766c4973243d2febf422333b4c73b548eb1c74fb2c5ccaf3da13e75a027"}
Aug 04 23:08:49.528667 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.528621    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:08:49.535849 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.535828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:08:49.540363 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.540347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.542699 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.542678    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ec06a8\""
Aug 04 23:08:49.542808 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.542738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ec06a8-dockercfg-d8q6f\""
Aug 04 23:08:49.549331 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.549312    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:08:49.568697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.568674    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:08:49.572107 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.572092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.582822 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.582801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:08:49.602608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.602586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkwf\" (UniqueName: \"kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.602762 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.602740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.703683 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.703652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5q8\" (UniqueName: \"kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.703840 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.703696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkwf\" (UniqueName: \"kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.703840 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.703797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.703961 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.703841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.704517 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.704499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.711309 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.711282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkwf\" (UniqueName: \"kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf\") pod \"payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.804495 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.804417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.804495 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.804483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5q8\" (UniqueName: \"kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.805086 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.805069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.812091 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.812073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5q8\" (UniqueName: \"kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8\") pod \"payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:49.852143 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.852123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:49.882697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:49.882675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:50.011910 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.011827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:08:50.015805 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:08:50.015774    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab24551_fff8_426a_832b_c000aebf2eea.slice/crio-7f7873a1e1213d89560f6a7286ff048eb49d8345931c714ac9fe06cfe0e0a8af WatchSource:0}: Error finding container 7f7873a1e1213d89560f6a7286ff048eb49d8345931c714ac9fe06cfe0e0a8af: Status 404 returned error can't find the container with id 7f7873a1e1213d89560f6a7286ff048eb49d8345931c714ac9fe06cfe0e0a8af
Aug 04 23:08:50.048533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.048495    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:08:50.053056 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:08:50.053026    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2bd1eb_2fda_410e_94ad_498279bf6dce.slice/crio-0eaa203c45a7ed8b5f13aa75ec1e10f945b7764a7d3a75c02a5bef0ea3880fdb WatchSource:0}: Error finding container 0eaa203c45a7ed8b5f13aa75ec1e10f945b7764a7d3a75c02a5bef0ea3880fdb: Status 404 returned error can't find the container with id 0eaa203c45a7ed8b5f13aa75ec1e10f945b7764a7d3a75c02a5bef0ea3880fdb
Aug 04 23:08:50.493956 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.493918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" event={"ID":"f4f89345-6a74-43a6-97e1-24854a4b3f25","Type":"ContainerStarted","Data":"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"}
Aug 04 23:08:50.494150 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.494037    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:08:50.495484 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.495454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" event={"ID":"8a2bd1eb-2fda-410e-94ad-498279bf6dce","Type":"ContainerStarted","Data":"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"}
Aug 04 23:08:50.495620 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.495510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:08:50.495620 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.495536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" event={"ID":"8a2bd1eb-2fda-410e-94ad-498279bf6dce","Type":"ContainerStarted","Data":"0eaa203c45a7ed8b5f13aa75ec1e10f945b7764a7d3a75c02a5bef0ea3880fdb"}
Aug 04 23:08:50.497144 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.497115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" event={"ID":"7ab24551-fff8-426a-832b-c000aebf2eea","Type":"ContainerStarted","Data":"0c69c23563c40da6dd310d42bda662f6ec2acfba962d01c4044e584a5b453bba"}
Aug 04 23:08:50.497244 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.497154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" event={"ID":"7ab24551-fff8-426a-832b-c000aebf2eea","Type":"ContainerStarted","Data":"7f7873a1e1213d89560f6a7286ff048eb49d8345931c714ac9fe06cfe0e0a8af"}
Aug 04 23:08:50.497244 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.497180    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:08:50.509585 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.509534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" podStartSLOduration=2.004357983 podStartE2EDuration="2.509516805s" totalImagesPullingTime="505.158822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:08:50.508796447 +0000 UTC m=+2868.050289622" watchObservedRunningTime="2026-08-04 23:08:50.509516805 +0000 UTC m=+2868.051009980"
Aug 04 23:08:50.524162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.524118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" podStartSLOduration=1.5241057690000002 podStartE2EDuration="1.524105769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:08:50.52298405 +0000 UTC m=+2868.064477251" watchObservedRunningTime="2026-08-04 23:08:50.524105769 +0000 UTC m=+2868.065598944"
Aug 04 23:08:50.537645 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:50.537605    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" podStartSLOduration=1.537593824 podStartE2EDuration="1.537593824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:08:50.536116456 +0000 UTC m=+2868.077609643" watchObservedRunningTime="2026-08-04 23:08:50.537593824 +0000 UTC m=+2868.079086997"
Aug 04 23:08:56.508080 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:08:56.508046    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:09:01.504439 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:01.504409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:09:01.504837 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:01.504475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:09:12.811564 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:12.811526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:13.153093 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:13.153061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:13.458789 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:13.458696    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:13.557443 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:13.557412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:16.348537 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:16.348505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:16.402289 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:16.402260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:16.674573 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:16.674547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:16.777346 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:16.777313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:17.816110 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:17.816077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:17.883882 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:17.883852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:18.349731 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.349695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:18.450159 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.450126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:18.533995 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.533968    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:09:18.537701 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.537684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.540253 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.540234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-55909a\""
Aug 04 23:09:18.540474 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.540445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-55909a-dockercfg-tfsgl\""
Aug 04 23:09:18.545594 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.545573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:09:18.661411 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.661382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.661595 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.661500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzn6d\" (UniqueName: \"kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.762430 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.762397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.762591 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.762526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzn6d\" (UniqueName: \"kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.762948 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.762929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.771524 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.771500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzn6d\" (UniqueName: \"kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d\") pod \"payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.848864 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.848836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:18.987870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:18.987845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:09:18.990150 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:09:18.990114    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fbf540_8da9_45af_a200_1f6393bf9c63.slice/crio-61fc680def433f2d7bd5f2fc53693d716875a8062af7909d1947e7e986374267 WatchSource:0}: Error finding container 61fc680def433f2d7bd5f2fc53693d716875a8062af7909d1947e7e986374267: Status 404 returned error can't find the container with id 61fc680def433f2d7bd5f2fc53693d716875a8062af7909d1947e7e986374267
Aug 04 23:09:19.645167 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:19.645112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" event={"ID":"64fbf540-8da9-45af-a200-1f6393bf9c63","Type":"ContainerStarted","Data":"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"}
Aug 04 23:09:19.645341 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:19.645180    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" event={"ID":"64fbf540-8da9-45af-a200-1f6393bf9c63","Type":"ContainerStarted","Data":"61fc680def433f2d7bd5f2fc53693d716875a8062af7909d1947e7e986374267"}
Aug 04 23:09:19.645341 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:19.645214    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:19.660800 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:19.660505    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" podStartSLOduration=1.660485536 podStartE2EDuration="1.660485536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:09:19.659797494 +0000 UTC m=+2897.201290668" watchObservedRunningTime="2026-08-04 23:09:19.660485536 +0000 UTC m=+2897.201978711"
Aug 04 23:09:30.651667 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:30.651639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:09:41.695525 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:41.695490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:09:47.287251 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:09:47.287219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:10:08.096588 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.096542    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:10:08.101194 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.101169    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.104351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.104327    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"kube-root-ca.crt\""
Aug 04 23:10:08.104535 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.104397    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"auth-test-model-55909a-kserve-self-signed-certs\""
Aug 04 23:10:08.104646 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.104416    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"openshift-service-ca.crt\""
Aug 04 23:10:08.104716 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.104416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"default-dockercfg-ksblg\""
Aug 04 23:10:08.110757 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.110734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:10:08.130504 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.130477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:10:08.195851 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.195821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgb8g\" (UniqueName: \"kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.196016 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.195872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.196016 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.195945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.196016 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.195989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.196124 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.196068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.196124 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.196096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297381 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297479 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297479 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297479 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgb8g\" (UniqueName: \"kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297633 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297692 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297791 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297925 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.297925 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.297821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.299939 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.299918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.300091 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.300075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.306018 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.305999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgb8g\" (UniqueName: \"kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g\") pod \"auth-test-model-55909a-kserve-5d4447789f-jqnqq\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") " pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.412712 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.412691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:08.545762 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.545737    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:10:08.547744 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:10:08.547713    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0737806a_1a11_49d6_b9f1_0361e0a81391.slice/crio-0f2e88a7d627a0eed473dd4093b38000ec71b1bb0665385059291cc856643ea7 WatchSource:0}: Error finding container 0f2e88a7d627a0eed473dd4093b38000ec71b1bb0665385059291cc856643ea7: Status 404 returned error can't find the container with id 0f2e88a7d627a0eed473dd4093b38000ec71b1bb0665385059291cc856643ea7
Aug 04 23:10:08.888322 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.888293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" event={"ID":"0737806a-1a11-49d6-b9f1-0361e0a81391","Type":"ContainerStarted","Data":"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"}
Aug 04 23:10:08.888494 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.888333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" event={"ID":"0737806a-1a11-49d6-b9f1-0361e0a81391","Type":"ContainerStarted","Data":"0f2e88a7d627a0eed473dd4093b38000ec71b1bb0665385059291cc856643ea7"}
Aug 04 23:10:08.904829 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:08.904783    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" podStartSLOduration=0.904770326 podStartE2EDuration="904.770326ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:10:08.902555568 +0000 UTC m=+2946.444048742" watchObservedRunningTime="2026-08-04 23:10:08.904770326 +0000 UTC m=+2946.446263499"
Aug 04 23:10:18.413813 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:18.413753    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:18.414351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:18.413848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:18.426650 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:18.426622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:18.952247 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:18.952216    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:10:39.483174 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:10:39.483086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:11:03.132105 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:11:03.132079    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:11:03.140533 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:11:03.140513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:11:08.034889 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:11:08.034856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:11:09.580559 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:11:09.580518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:11:24.365338 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:11:24.365300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:12:59.919218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:12:59.919182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:12:59.970191 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:12:59.970157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:13:17.495749 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:17.495714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:13:18.372815 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:18.372780    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:13:18.391884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:18.391850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:13:30.457705 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:30.457667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:13:33.393111 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.393069    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" podUID="0737806a-1a11-49d6-b9f1-0361e0a81391" containerName="main" containerID="cri-o://0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc" gracePeriod=45
Aug 04 23:13:33.641815 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.641792    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:13:33.726249 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726187    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726249 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgb8g\" (UniqueName: \"kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726249 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726519 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726519 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726519 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726359    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache\") pod \"0737806a-1a11-49d6-b9f1-0361e0a81391\" (UID: \"0737806a-1a11-49d6-b9f1-0361e0a81391\") "
Aug 04 23:13:33.726519 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726502    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:13:33.726720 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home" (OuterVolumeSpecName: "home") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:13:33.726720 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726647    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache" (OuterVolumeSpecName: "model-cache") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:13:33.726825 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726736    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.726825 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726749    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.726825 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.726758    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.728541 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.728513    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm" (OuterVolumeSpecName: "dshm") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:13:33.728673 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.728564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g" (OuterVolumeSpecName: "kube-api-access-tgb8g") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "kube-api-access-tgb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:13:33.728673 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.728570    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0737806a-1a11-49d6-b9f1-0361e0a81391" (UID: "0737806a-1a11-49d6-b9f1-0361e0a81391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:13:33.828065 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.828040    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737806a-1a11-49d6-b9f1-0361e0a81391-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.828065 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.828061    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737806a-1a11-49d6-b9f1-0361e0a81391-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.828212 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.828070    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgb8g\" (UniqueName: \"kubernetes.io/projected/0737806a-1a11-49d6-b9f1-0361e0a81391-kube-api-access-tgb8g\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:13:33.904050 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.904009    2582 generic.go:356] "Generic (PLEG): container finished" podID="0737806a-1a11-49d6-b9f1-0361e0a81391" containerID="0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc" exitCode=0
Aug 04 23:13:33.904182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.904067    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"
Aug 04 23:13:33.904182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.904101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" event={"ID":"0737806a-1a11-49d6-b9f1-0361e0a81391","Type":"ContainerDied","Data":"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"}
Aug 04 23:13:33.904182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.904149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq" event={"ID":"0737806a-1a11-49d6-b9f1-0361e0a81391","Type":"ContainerDied","Data":"0f2e88a7d627a0eed473dd4093b38000ec71b1bb0665385059291cc856643ea7"}
Aug 04 23:13:33.904182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.904167    2582 scope.go:117] "RemoveContainer" containerID="0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"
Aug 04 23:13:33.913908 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.913874    2582 scope.go:117] "RemoveContainer" containerID="0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"
Aug 04 23:13:33.914143 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:13:33.914126    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc\": container with ID starting with 0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc not found: ID does not exist" containerID="0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"
Aug 04 23:13:33.914195 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.914149    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc"} err="failed to get container status \"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc\": rpc error: code = NotFound desc = could not find container \"0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc\": container with ID starting with 0975ea2d1dd943065028c197c4331da32941576326fc9bd9c2d84e8fbf2748dc not found: ID does not exist"
Aug 04 23:13:33.924400 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.924378    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:13:33.927367 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:33.927338    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/auth-test-model-55909a-kserve-5d4447789f-jqnqq"]
Aug 04 23:13:35.009966 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:35.009936    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0737806a-1a11-49d6-b9f1-0361e0a81391" path="/var/lib/kubelet/pods/0737806a-1a11-49d6-b9f1-0361e0a81391/volumes"
Aug 04 23:13:44.095077 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.095034    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:13:44.095666 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.095640    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0737806a-1a11-49d6-b9f1-0361e0a81391" containerName="main"
Aug 04 23:13:44.095666 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.095662    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0737806a-1a11-49d6-b9f1-0361e0a81391" containerName="main"
Aug 04 23:13:44.095834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.095809    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0737806a-1a11-49d6-b9f1-0361e0a81391" containerName="main"
Aug 04 23:13:44.099147 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.099127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.102332 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.102310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec06a8\"/\"default-dockercfg-g4dlq\""
Aug 04 23:13:44.102448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.102310    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec06a8\"/\"openshift-service-ca.crt\""
Aug 04 23:13:44.102448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.102348    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec06a8\"/\"kube-root-ca.crt\""
Aug 04 23:13:44.102448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.102312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec06a8\"/\"sub-test-model-ec06a8-kserve-self-signed-certs\""
Aug 04 23:13:44.107222 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.107192    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:13:44.126395 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.126361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:13:44.216737 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.216712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.216856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.216751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.216856 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.216777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtk9\" (UniqueName: \"kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.217021 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.216849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.217021 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.216967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.217122 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.217024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318311 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtk9\" (UniqueName: \"kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318690 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318759 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.318823 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.318786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.320752 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.320733    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.321115 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.321099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.325427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.325408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtk9\" (UniqueName: \"kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9\") pod \"sub-test-model-ec06a8-kserve-84c6dcc97c-8694l\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") " pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.410831 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.410800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:44.542639 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.542614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:13:44.544643 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:13:44.544615    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbaba83_fdda_4634_940e_ca9d9d558f90.slice/crio-8e0aca7e8485858c31b7b7703dcac2f475070911904590570064ac2137a1242e WatchSource:0}: Error finding container 8e0aca7e8485858c31b7b7703dcac2f475070911904590570064ac2137a1242e: Status 404 returned error can't find the container with id 8e0aca7e8485858c31b7b7703dcac2f475070911904590570064ac2137a1242e
Aug 04 23:13:44.958500 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.958461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" event={"ID":"dfbaba83-fdda-4634-940e-ca9d9d558f90","Type":"ContainerStarted","Data":"fa5e348d4754ced19fa211bfbb84a0085c0166701073926840115f9088ad61a3"}
Aug 04 23:13:44.958500 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.958506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" event={"ID":"dfbaba83-fdda-4634-940e-ca9d9d558f90","Type":"ContainerStarted","Data":"8e0aca7e8485858c31b7b7703dcac2f475070911904590570064ac2137a1242e"}
Aug 04 23:13:44.973176 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:44.973127    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" podStartSLOduration=0.973109804 podStartE2EDuration="973.109804ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:13:44.972424572 +0000 UTC m=+3162.513917744" watchObservedRunningTime="2026-08-04 23:13:44.973109804 +0000 UTC m=+3162.514602979"
Aug 04 23:13:54.411535 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:54.411492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:54.412014 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:54.411560    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:54.424115 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:54.424093    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:55.027230 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:55.027197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:13:59.546862 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:13:59.546826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:15:09.316919 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:09.316827    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:15:10.099257 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:10.099224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:15:10.515049 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:10.515012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:15:24.338955 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.338883    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" podUID="dfbaba83-fdda-4634-940e-ca9d9d558f90" containerName="main" containerID="cri-o://fa5e348d4754ced19fa211bfbb84a0085c0166701073926840115f9088ad61a3" gracePeriod=45
Aug 04 23:15:24.469958 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.469914    2582 generic.go:356] "Generic (PLEG): container finished" podID="dfbaba83-fdda-4634-940e-ca9d9d558f90" containerID="fa5e348d4754ced19fa211bfbb84a0085c0166701073926840115f9088ad61a3" exitCode=0
Aug 04 23:15:24.470106 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.469970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" event={"ID":"dfbaba83-fdda-4634-940e-ca9d9d558f90","Type":"ContainerDied","Data":"fa5e348d4754ced19fa211bfbb84a0085c0166701073926840115f9088ad61a3"}
Aug 04 23:15:24.592618 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.592561    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:15:24.714610 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.714789 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714621    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.714789 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714707    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.714789 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714730    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.714789 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.715044 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714798    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgtk9\" (UniqueName: \"kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9\") pod \"dfbaba83-fdda-4634-940e-ca9d9d558f90\" (UID: \"dfbaba83-fdda-4634-940e-ca9d9d558f90\") "
Aug 04 23:15:24.715044 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:15:24.715044 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.714995    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:15:24.715166 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.715053    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home" (OuterVolumeSpecName: "home") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:15:24.715208 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.715169    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:24.715208 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.715186    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:24.715208 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.715195    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:24.717027 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.717002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm" (OuterVolumeSpecName: "dshm") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:15:24.717027 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.717001    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:15:24.717170 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.717051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9" (OuterVolumeSpecName: "kube-api-access-fgtk9") pod "dfbaba83-fdda-4634-940e-ca9d9d558f90" (UID: "dfbaba83-fdda-4634-940e-ca9d9d558f90"). InnerVolumeSpecName "kube-api-access-fgtk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:15:24.816456 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.816429    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbaba83-fdda-4634-940e-ca9d9d558f90-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:24.816456 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.816453    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtk9\" (UniqueName: \"kubernetes.io/projected/dfbaba83-fdda-4634-940e-ca9d9d558f90-kube-api-access-fgtk9\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:24.816597 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:24.816463    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbaba83-fdda-4634-940e-ca9d9d558f90-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:15:25.476987 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:25.476952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l" event={"ID":"dfbaba83-fdda-4634-940e-ca9d9d558f90","Type":"ContainerDied","Data":"8e0aca7e8485858c31b7b7703dcac2f475070911904590570064ac2137a1242e"}
Aug 04 23:15:25.477397 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:25.477008    2582 scope.go:117] "RemoveContainer" containerID="fa5e348d4754ced19fa211bfbb84a0085c0166701073926840115f9088ad61a3"
Aug 04 23:15:25.477397 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:25.476964    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"
Aug 04 23:15:25.491248 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:25.491225    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:15:25.496398 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:25.496379    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ec06a8/sub-test-model-ec06a8-kserve-84c6dcc97c-8694l"]
Aug 04 23:15:27.011056 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:27.011010    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbaba83-fdda-4634-940e-ca9d9d558f90" path="/var/lib/kubelet/pods/dfbaba83-fdda-4634-940e-ca9d9d558f90/volumes"
Aug 04 23:15:30.924186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:15:30.924146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:16:03.163113 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:03.163083    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:16:03.171815 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:03.171782    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:16:12.999668 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:12.999635    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:16:13.000076 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.000062    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbaba83-fdda-4634-940e-ca9d9d558f90" containerName="main"
Aug 04 23:16:13.000120 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.000079    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbaba83-fdda-4634-940e-ca9d9d558f90" containerName="main"
Aug 04 23:16:13.000172 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.000161    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbaba83-fdda-4634-940e-ca9d9d558f90" containerName="main"
Aug 04 23:16:13.003265 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.003246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.005500 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.005477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"kube-root-ca.crt\""
Aug 04 23:16:13.005675 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.005510    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"openshift-service-ca.crt\""
Aug 04 23:16:13.006290 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.006266    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"rate-test-model-55909a-kserve-self-signed-certs\""
Aug 04 23:16:13.006399 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.006274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-55909a\"/\"default-dockercfg-ksblg\""
Aug 04 23:16:13.013375 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.013354    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:16:13.044380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.044326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:16:13.134760 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.134872 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659vx\" (UniqueName: \"kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.134872 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.134996 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.134996 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.134996 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.134989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.235792 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.235771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.235927 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.235824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.235927 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.235847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.235927 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.235873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.236087 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.236033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659vx\" (UniqueName: \"kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.236150 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.236121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.236207 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.236173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.236266 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.236216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.236322 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.236284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.238286 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.238261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.238434 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.238418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.243200 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.243180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-659vx\" (UniqueName: \"kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx\") pod \"rate-test-model-55909a-kserve-7b994b5d87-z2cnt\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") " pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.315107 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.315051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:13.445171 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.445144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:16:13.446954 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:16:13.446926    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4efe8a20_b805_40ae_8875_53d0707e76ca.slice/crio-8f557cae72b64831360c0d1a9f3aa900441f7d4188285131854bfc1e7f620292 WatchSource:0}: Error finding container 8f557cae72b64831360c0d1a9f3aa900441f7d4188285131854bfc1e7f620292: Status 404 returned error can't find the container with id 8f557cae72b64831360c0d1a9f3aa900441f7d4188285131854bfc1e7f620292
Aug 04 23:16:13.725013 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.724971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" event={"ID":"4efe8a20-b805-40ae-8875-53d0707e76ca","Type":"ContainerStarted","Data":"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"}
Aug 04 23:16:13.725201 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.725028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" event={"ID":"4efe8a20-b805-40ae-8875-53d0707e76ca","Type":"ContainerStarted","Data":"8f557cae72b64831360c0d1a9f3aa900441f7d4188285131854bfc1e7f620292"}
Aug 04 23:16:13.743391 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:13.743333    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" podStartSLOduration=1.74331815 podStartE2EDuration="1.74331815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:16:13.740699987 +0000 UTC m=+3311.282193161" watchObservedRunningTime="2026-08-04 23:16:13.74331815 +0000 UTC m=+3311.284811324"
Aug 04 23:16:22.669512 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:22.669474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:16:23.315568 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:23.315532    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:23.315786 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:23.315589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:23.328632 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:23.328606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:16:23.785047 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:16:23.785017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:17:38.810839 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:38.810810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:17:39.872035 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:39.872003    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:17:39.890861 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:39.890833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:17:54.892675 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:54.892552    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" podUID="4efe8a20-b805-40ae-8875-53d0707e76ca" containerName="main" containerID="cri-o://98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0" gracePeriod=45
Aug 04 23:17:55.146772 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.146717    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:17:55.214449 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214424    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214458    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-659vx\" (UniqueName: \"kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214591    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm\") pod \"4efe8a20-b805-40ae-8875-53d0707e76ca\" (UID: \"4efe8a20-b805-40ae-8875-53d0707e76ca\") "
Aug 04 23:17:55.214887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214674    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:17:55.214887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214839    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home" (OuterVolumeSpecName: "home") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:17:55.214887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214857    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:17:55.215020 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.214997    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:55.215020 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.215015    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:55.215095 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.215027    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:55.216750 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.216727    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm" (OuterVolumeSpecName: "dshm") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:17:55.216750 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.216725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:17:55.216936 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.216806    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx" (OuterVolumeSpecName: "kube-api-access-659vx") pod "4efe8a20-b805-40ae-8875-53d0707e76ca" (UID: "4efe8a20-b805-40ae-8875-53d0707e76ca"). InnerVolumeSpecName "kube-api-access-659vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:17:55.232076 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.232049    2582 generic.go:356] "Generic (PLEG): container finished" podID="4efe8a20-b805-40ae-8875-53d0707e76ca" containerID="98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0" exitCode=0
Aug 04 23:17:55.232163 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.232091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"
Aug 04 23:17:55.232163 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.232126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" event={"ID":"4efe8a20-b805-40ae-8875-53d0707e76ca","Type":"ContainerDied","Data":"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"}
Aug 04 23:17:55.232247 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.232167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt" event={"ID":"4efe8a20-b805-40ae-8875-53d0707e76ca","Type":"ContainerDied","Data":"8f557cae72b64831360c0d1a9f3aa900441f7d4188285131854bfc1e7f620292"}
Aug 04 23:17:55.232247 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.232184    2582 scope.go:117] "RemoveContainer" containerID="98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"
Aug 04 23:17:55.241167 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.241152    2582 scope.go:117] "RemoveContainer" containerID="98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"
Aug 04 23:17:55.241409 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:17:55.241390    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0\": container with ID starting with 98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0 not found: ID does not exist" containerID="98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"
Aug 04 23:17:55.241451 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.241418    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0"} err="failed to get container status \"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0\": rpc error: code = NotFound desc = could not find container \"98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0\": container with ID starting with 98efb3efb4f5cf542f3164cfbdc775d64975d4374b61b46d87d31dd734369ac0 not found: ID does not exist"
Aug 04 23:17:55.254327 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.254308    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:17:55.257506 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.257487    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-55909a/rate-test-model-55909a-kserve-7b994b5d87-z2cnt"]
Aug 04 23:17:55.316181 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.316156    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-659vx\" (UniqueName: \"kubernetes.io/projected/4efe8a20-b805-40ae-8875-53d0707e76ca-kube-api-access-659vx\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:55.316269 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.316181    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4efe8a20-b805-40ae-8875-53d0707e76ca-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:55.316269 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:55.316192    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4efe8a20-b805-40ae-8875-53d0707e76ca-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:17:57.010738 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:57.010693    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4efe8a20-b805-40ae-8875-53d0707e76ca" path="/var/lib/kubelet/pods/4efe8a20-b805-40ae-8875-53d0707e76ca/volumes"
Aug 04 23:17:59.060958 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:59.060923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:17:59.353788 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:59.353702    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:17:59.454570 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:17:59.454536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:01.840364 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:01.840323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:01.943427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:01.943391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:02.935558 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.935530    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:18:02.935962 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.935949    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efe8a20-b805-40ae-8875-53d0707e76ca" containerName="main"
Aug 04 23:18:02.936012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.935964    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efe8a20-b805-40ae-8875-53d0707e76ca" containerName="main"
Aug 04 23:18:02.936055 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.936047    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efe8a20-b805-40ae-8875-53d0707e76ca" containerName="main"
Aug 04 23:18:02.940561 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.940542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:02.942746 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.942724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d1b011-serving-cert\""
Aug 04 23:18:02.948455 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.948434    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:18:02.965528 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.965504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:02.984410 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.984387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpbc\" (UniqueName: \"kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:02.984520 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.984440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:02.984520 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:02.984470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.085188 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.085150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpbc\" (UniqueName: \"kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.085377 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.085215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.085377 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.085263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.085515 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:18:03.085385    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-serving-cert: secret "maas-api-e2e-ipp-a-d1b011-serving-cert" not found
Aug 04 23:18:03.085515 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:18:03.085459    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls podName:625e245a-1127-428b-aca4-8aa22eafc4d1 nodeName:}" failed. No retries permitted until 2026-08-04 23:18:03.585437112 +0000 UTC m=+3421.126930270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls") pod "maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" (UID: "625e245a-1127-428b-aca4-8aa22eafc4d1") : secret "maas-api-e2e-ipp-a-d1b011-serving-cert" not found
Aug 04 23:18:03.085964 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.085944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.093870 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.093839    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpbc\" (UniqueName: \"kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.220199 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.220117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:03.443139 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.443104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:03.545289 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.545216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:03.575813 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.575780    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:18:03.581360 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.581335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.583524 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.583503    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d1b011-dockercfg-ht9vk\""
Aug 04 23:18:03.583809 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.583791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1b011\""
Aug 04 23:18:03.589203 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.589177    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:18:03.591385 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.591359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.593974 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.593954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.692048 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.692021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.692048 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.692050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dzh\" (UniqueName: \"kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.793054 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.793023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.793054 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.793056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dzh\" (UniqueName: \"kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.793763 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.793727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.800221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.800155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dzh\" (UniqueName: \"kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh\") pod \"payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.854554 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.854531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:03.894080 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.893744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:03.996453 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:03.996167    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:18:03.999679 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:18:03.999654    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625e245a_1127_428b_aca4_8aa22eafc4d1.slice/crio-4eecdeab791c7310a3e74299334b9c2e2fc841dcd41efaa095e1b92197b2fc28 WatchSource:0}: Error finding container 4eecdeab791c7310a3e74299334b9c2e2fc841dcd41efaa095e1b92197b2fc28: Status 404 returned error can't find the container with id 4eecdeab791c7310a3e74299334b9c2e2fc841dcd41efaa095e1b92197b2fc28
Aug 04 23:18:04.001380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.001359    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 23:18:04.042381 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.042302    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:18:04.044826 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:18:04.044804    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c4a016_81fe_495a_8a62_b79101ffaf74.slice/crio-9a88c690fc7374c66c3d8ef80e4cc3a9fe30602649defd497f0ddb8e157d0068 WatchSource:0}: Error finding container 9a88c690fc7374c66c3d8ef80e4cc3a9fe30602649defd497f0ddb8e157d0068: Status 404 returned error can't find the container with id 9a88c690fc7374c66c3d8ef80e4cc3a9fe30602649defd497f0ddb8e157d0068
Aug 04 23:18:04.279506 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.279468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" event={"ID":"625e245a-1127-428b-aca4-8aa22eafc4d1","Type":"ContainerStarted","Data":"4eecdeab791c7310a3e74299334b9c2e2fc841dcd41efaa095e1b92197b2fc28"}
Aug 04 23:18:04.281251 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.281223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" event={"ID":"91c4a016-81fe-495a-8a62-b79101ffaf74","Type":"ContainerStarted","Data":"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"}
Aug 04 23:18:04.281391 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.281268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" event={"ID":"91c4a016-81fe-495a-8a62-b79101ffaf74","Type":"ContainerStarted","Data":"9a88c690fc7374c66c3d8ef80e4cc3a9fe30602649defd497f0ddb8e157d0068"}
Aug 04 23:18:04.281391 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.281340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:04.296825 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:04.296772    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" podStartSLOduration=1.29675241 podStartE2EDuration="1.29675241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:18:04.295499923 +0000 UTC m=+3421.836993098" watchObservedRunningTime="2026-08-04 23:18:04.29675241 +0000 UTC m=+3421.838245581"
Aug 04 23:18:05.288451 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:05.288393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" event={"ID":"625e245a-1127-428b-aca4-8aa22eafc4d1","Type":"ContainerStarted","Data":"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"}
Aug 04 23:18:05.308225 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:05.308176    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" podStartSLOduration=2.850605544 podStartE2EDuration="3.308160417s" totalImagesPullingTime="457.554873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:18:05.30650129 +0000 UTC m=+3422.847994463" watchObservedRunningTime="2026-08-04 23:18:05.308160417 +0000 UTC m=+3422.849653588"
Aug 04 23:18:06.294343 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:06.294308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:12.305162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:12.305132    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:18:15.291223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:15.291189    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:18:26.841199 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:26.841156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:27.540231 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:27.540186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:27.844492 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:27.844418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:27.955251 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:27.955212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:30.192571 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:30.192541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:30.963637 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:30.963600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:31.063587 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:31.063549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:32.065259 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.065227    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:18:32.069583 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.069558    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.072465 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.072441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-edf378-serving-cert\""
Aug 04 23:18:32.080084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.080060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:18:32.096108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.096077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:32.158398 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.158369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.158605 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.158440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjv5\" (UniqueName: \"kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.158605 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.158495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.259723 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.259690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.259926 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.259747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.259926 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.259810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjv5\" (UniqueName: \"kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.260020 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:18:32.259931    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-serving-cert: secret "maas-api-e2e-ipp-b-edf378-serving-cert" not found
Aug 04 23:18:32.260020 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:18:32.259996    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls podName:028dd80e-7aa3-4116-80d1-a8366e2b17ee nodeName:}" failed. No retries permitted until 2026-08-04 23:18:32.759979439 +0000 UTC m=+3450.301472592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls") pod "maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" (UID: "028dd80e-7aa3-4116-80d1-a8366e2b17ee") : secret "maas-api-e2e-ipp-b-edf378-serving-cert" not found
Aug 04 23:18:32.260369 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.260349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.271292 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.271270    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjv5\" (UniqueName: \"kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.591089 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.591056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:32.697509 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.697475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:32.764948 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.764884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.767542 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.767523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:32.768403 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.768379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:18:32.982371 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:32.982334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:33.137523 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:33.137496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:18:33.139435 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:18:33.139401    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028dd80e_7aa3_4116_80d1_a8366e2b17ee.slice/crio-b4d8509c8dac5e29e888c8e582c745de98f8c9e49df851f5a48e406373f42690 WatchSource:0}: Error finding container b4d8509c8dac5e29e888c8e582c745de98f8c9e49df851f5a48e406373f42690: Status 404 returned error can't find the container with id b4d8509c8dac5e29e888c8e582c745de98f8c9e49df851f5a48e406373f42690
Aug 04 23:18:33.428845 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:33.428809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" event={"ID":"028dd80e-7aa3-4116-80d1-a8366e2b17ee","Type":"ContainerStarted","Data":"b4d8509c8dac5e29e888c8e582c745de98f8c9e49df851f5a48e406373f42690"}
Aug 04 23:18:34.435087 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:34.435048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" event={"ID":"028dd80e-7aa3-4116-80d1-a8366e2b17ee","Type":"ContainerStarted","Data":"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"}
Aug 04 23:18:34.435494 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:34.435165    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:34.450235 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:34.450180    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" podStartSLOduration=1.806833227 podStartE2EDuration="2.450162748s" totalImagesPullingTime="643.329521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:18:34.44984928 +0000 UTC m=+3451.991342455" watchObservedRunningTime="2026-08-04 23:18:34.450162748 +0000 UTC m=+3451.991655962"
Aug 04 23:18:40.444715 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:40.444687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:18:55.971666 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:18:55.971622    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:19:02.914681 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:02.914622    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:19:35.900454 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.900358    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:19:35.904229 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.904208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.906298 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.906275    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d1b011\"/\"openshift-service-ca.crt\""
Aug 04 23:19:35.907144 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.907106    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d1b011\"/\"kube-root-ca.crt\""
Aug 04 23:19:35.907266 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.907168    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d1b011\"/\"ipp-route-d1b011-kserve-self-signed-certs\""
Aug 04 23:19:35.907266 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.907168    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d1b011\"/\"default-dockercfg-q65zb\""
Aug 04 23:19:35.910223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.910204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:19:35.922632 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.922602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:19:35.960946 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.960918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.961084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.960955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.961084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.960978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.961084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.960998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.961084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.961025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2vv\" (UniqueName: \"kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:35.961084 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:35.961047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062396 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062573 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062573 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062573 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062832 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.062936 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2vv\" (UniqueName: \"kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.063012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.062977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.063012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.063000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.063125 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.063105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.065137 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.065114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.065137 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.065132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.069884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.069866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2vv\" (UniqueName: \"kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv\") pod \"ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") " pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.215647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.215568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:36.347932 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.347885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:19:36.349942 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:19:36.349886    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd40264_a810_438e_8cc0_df02308cc8cc.slice/crio-36d99d4e3b02511daf0a33de73a92849d5b82d1b7f793bfa69bc6d214df6a9bc WatchSource:0}: Error finding container 36d99d4e3b02511daf0a33de73a92849d5b82d1b7f793bfa69bc6d214df6a9bc: Status 404 returned error can't find the container with id 36d99d4e3b02511daf0a33de73a92849d5b82d1b7f793bfa69bc6d214df6a9bc
Aug 04 23:19:36.751715 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.751665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" event={"ID":"1dd40264-a810-438e-8cc0-df02308cc8cc","Type":"ContainerStarted","Data":"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"}
Aug 04 23:19:36.751715 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.751718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" event={"ID":"1dd40264-a810-438e-8cc0-df02308cc8cc","Type":"ContainerStarted","Data":"36d99d4e3b02511daf0a33de73a92849d5b82d1b7f793bfa69bc6d214df6a9bc"}
Aug 04 23:19:36.767633 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:36.767583    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" podStartSLOduration=1.76756969 podStartE2EDuration="1.76756969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:19:36.765729028 +0000 UTC m=+3514.307222202" watchObservedRunningTime="2026-08-04 23:19:36.76756969 +0000 UTC m=+3514.309062863"
Aug 04 23:19:39.600136 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:39.600092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:19:46.215785 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.215752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:46.216206 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.215804    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:46.228422 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.228392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:19:46.659182 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.659149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:19:46.759758 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.759732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:19:46.814027 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:19:46.814004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:20:13.508697 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:13.508651    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww_91c4a016-81fe-495a-8a62-b79101ffaf74/payload-processing/0.log"
Aug 04 23:20:13.635035 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:13.635010    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:20:14.048027 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:14.047997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:14.348665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:14.348586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:14.448753 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:14.448718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:17.538455 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:17.538411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:17.640112 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:17.640072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:18.657928 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:18.657878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:18.870587 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:18.870554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:19.138677 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.138647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:19.235997 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.235967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:19.241364 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.241339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:19.245245 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.245230    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.247538 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.247518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2b5cc7\""
Aug 04 23:20:19.247653 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.247591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2b5cc7-dockercfg-bbptm\""
Aug 04 23:20:19.254487 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.254462    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:19.272500 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.272476    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:19.276099 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.276083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.289620 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.289593    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:19.348421 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.348388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.348570 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.348550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.348625 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.348581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txttr\" (UniqueName: \"kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.348625 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.348612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zq9\" (UniqueName: \"kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.449218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.449218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txttr\" (UniqueName: \"kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.449218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zq9\" (UniqueName: \"kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.449470 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.449886 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.449966 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.449873    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.461793 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.461765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txttr\" (UniqueName: \"kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.461887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.461791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zq9\" (UniqueName: \"kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9\") pod \"payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.557495 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.557466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.589291 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.589272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.707721 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.707688    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:19.708973 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:20:19.708940    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3fe89c_75c5_4385_868d_c5800adc4579.slice/crio-4543b1eea9047d7f9dfb2de68f4b9b4c5260cc162ff04f237eaaa98d3356f6b1 WatchSource:0}: Error finding container 4543b1eea9047d7f9dfb2de68f4b9b4c5260cc162ff04f237eaaa98d3356f6b1: Status 404 returned error can't find the container with id 4543b1eea9047d7f9dfb2de68f4b9b4c5260cc162ff04f237eaaa98d3356f6b1
Aug 04 23:20:19.742615 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.742589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:19.748022 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:20:19.747985    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6f6208_f842_425c_92c8_2f7193b50365.slice/crio-2116a120c9a371ed9ab31f154fd0a6d13022d7568ddabdba15a0f213b56c5fd7 WatchSource:0}: Error finding container 2116a120c9a371ed9ab31f154fd0a6d13022d7568ddabdba15a0f213b56c5fd7: Status 404 returned error can't find the container with id 2116a120c9a371ed9ab31f154fd0a6d13022d7568ddabdba15a0f213b56c5fd7
Aug 04 23:20:19.967374 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.967282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" event={"ID":"5c6f6208-f842-425c-92c8-2f7193b50365","Type":"ContainerStarted","Data":"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"}
Aug 04 23:20:19.967374 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.967357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" event={"ID":"5c6f6208-f842-425c-92c8-2f7193b50365","Type":"ContainerStarted","Data":"2116a120c9a371ed9ab31f154fd0a6d13022d7568ddabdba15a0f213b56c5fd7"}
Aug 04 23:20:19.967611 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.967387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:19.969066 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.969030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" event={"ID":"2e3fe89c-75c5-4385-868d-c5800adc4579","Type":"ContainerStarted","Data":"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"}
Aug 04 23:20:19.969186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.969075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" event={"ID":"2e3fe89c-75c5-4385-868d-c5800adc4579","Type":"ContainerStarted","Data":"4543b1eea9047d7f9dfb2de68f4b9b4c5260cc162ff04f237eaaa98d3356f6b1"}
Aug 04 23:20:19.969186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.969102    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:19.984111 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.984058    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" podStartSLOduration=0.984040125 podStartE2EDuration="984.040125ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:20:19.981815814 +0000 UTC m=+3557.523308989" watchObservedRunningTime="2026-08-04 23:20:19.984040125 +0000 UTC m=+3557.525533301"
Aug 04 23:20:19.995877 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:19.995828    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" podStartSLOduration=0.995811557 podStartE2EDuration="995.811557ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 23:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:20:19.99503314 +0000 UTC m=+3557.536526315" watchObservedRunningTime="2026-08-04 23:20:19.995811557 +0000 UTC m=+3557.537304733"
Aug 04 23:20:30.976973 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:30.976942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:30.977397 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:30.977011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:42.452151 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:42.452069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:49.155554 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.155524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:49.175017 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.174990    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:49.175251 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.175226    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" podUID="5c6f6208-f842-425c-92c8-2f7193b50365" containerName="payload-processing" containerID="cri-o://2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725" gracePeriod=30
Aug 04 23:20:49.185419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.184926    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:49.185419 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.185153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" podUID="2e3fe89c-75c5-4385-868d-c5800adc4579" containerName="payload-pre-processing" containerID="cri-o://13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501" gracePeriod=30
Aug 04 23:20:49.213666 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.213633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:49.271215 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.271184    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:49.331051 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.330962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:49.476077 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.476049    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:49.479151 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.479135    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:49.528631 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.528602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume\") pod \"2e3fe89c-75c5-4385-868d-c5800adc4579\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") "
Aug 04 23:20:49.528767 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.528681    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume\") pod \"5c6f6208-f842-425c-92c8-2f7193b50365\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") "
Aug 04 23:20:49.528767 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.528702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txttr\" (UniqueName: \"kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr\") pod \"2e3fe89c-75c5-4385-868d-c5800adc4579\" (UID: \"2e3fe89c-75c5-4385-868d-c5800adc4579\") "
Aug 04 23:20:49.528767 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.528724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9zq9\" (UniqueName: \"kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9\") pod \"5c6f6208-f842-425c-92c8-2f7193b50365\" (UID: \"5c6f6208-f842-425c-92c8-2f7193b50365\") "
Aug 04 23:20:49.529050 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.529023    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e3fe89c-75c5-4385-868d-c5800adc4579" (UID: "2e3fe89c-75c5-4385-868d-c5800adc4579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:20:49.529155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.529058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c6f6208-f842-425c-92c8-2f7193b50365" (UID: "5c6f6208-f842-425c-92c8-2f7193b50365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:20:49.530963 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.530935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9" (OuterVolumeSpecName: "kube-api-access-t9zq9") pod "5c6f6208-f842-425c-92c8-2f7193b50365" (UID: "5c6f6208-f842-425c-92c8-2f7193b50365"). InnerVolumeSpecName "kube-api-access-t9zq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:20:49.530963 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.530963    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr" (OuterVolumeSpecName: "kube-api-access-txttr") pod "2e3fe89c-75c5-4385-868d-c5800adc4579" (UID: "2e3fe89c-75c5-4385-868d-c5800adc4579"). InnerVolumeSpecName "kube-api-access-txttr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:20:49.629481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.629450    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c6f6208-f842-425c-92c8-2f7193b50365-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:20:49.629481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.629477    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txttr\" (UniqueName: \"kubernetes.io/projected/2e3fe89c-75c5-4385-868d-c5800adc4579-kube-api-access-txttr\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:20:49.629481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.629487    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zq9\" (UniqueName: \"kubernetes.io/projected/5c6f6208-f842-425c-92c8-2f7193b50365-kube-api-access-t9zq9\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:20:49.629698 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.629497    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3fe89c-75c5-4385-868d-c5800adc4579-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:20:49.656097 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:49.656067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:50.137168 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.137127    2582 generic.go:356] "Generic (PLEG): container finished" podID="5c6f6208-f842-425c-92c8-2f7193b50365" containerID="2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725" exitCode=0
Aug 04 23:20:50.137168 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.137168    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"
Aug 04 23:20:50.137426 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.137166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" event={"ID":"5c6f6208-f842-425c-92c8-2f7193b50365","Type":"ContainerDied","Data":"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"}
Aug 04 23:20:50.137426 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.137317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8" event={"ID":"5c6f6208-f842-425c-92c8-2f7193b50365","Type":"ContainerDied","Data":"2116a120c9a371ed9ab31f154fd0a6d13022d7568ddabdba15a0f213b56c5fd7"}
Aug 04 23:20:50.137426 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.137342    2582 scope.go:117] "RemoveContainer" containerID="2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"
Aug 04 23:20:50.138623 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.138592    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e3fe89c-75c5-4385-868d-c5800adc4579" containerID="13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501" exitCode=0
Aug 04 23:20:50.138747 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.138644    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"
Aug 04 23:20:50.138747 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.138675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" event={"ID":"2e3fe89c-75c5-4385-868d-c5800adc4579","Type":"ContainerDied","Data":"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"}
Aug 04 23:20:50.138747 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.138735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd" event={"ID":"2e3fe89c-75c5-4385-868d-c5800adc4579","Type":"ContainerDied","Data":"4543b1eea9047d7f9dfb2de68f4b9b4c5260cc162ff04f237eaaa98d3356f6b1"}
Aug 04 23:20:50.147780 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.147762    2582 scope.go:117] "RemoveContainer" containerID="2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"
Aug 04 23:20:50.148041 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:20:50.148018    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725\": container with ID starting with 2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725 not found: ID does not exist" containerID="2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"
Aug 04 23:20:50.148132 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.148051    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725"} err="failed to get container status \"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725\": rpc error: code = NotFound desc = could not find container \"2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725\": container with ID starting with 2fc6b511178c26321836890970df090d196b79bc1d3f42c2397e9a9c15d4d725 not found: ID does not exist"
Aug 04 23:20:50.148132 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.148073    2582 scope.go:117] "RemoveContainer" containerID="13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"
Aug 04 23:20:50.156078 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.156059    2582 scope.go:117] "RemoveContainer" containerID="13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"
Aug 04 23:20:50.156308 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:20:50.156292    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501\": container with ID starting with 13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501 not found: ID does not exist" containerID="13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"
Aug 04 23:20:50.156347 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.156314    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501"} err="failed to get container status \"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501\": rpc error: code = NotFound desc = could not find container \"13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501\": container with ID starting with 13a7ac03d36f22666b4c4b1700ae31de2d2b75e55787205131e9c3d655654501 not found: ID does not exist"
Aug 04 23:20:50.162046 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.162025    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:50.165519 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.165497    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b5cc7-5599dc54cb-jbkz8"]
Aug 04 23:20:50.174190 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.174171    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:50.177647 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:50.177626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b5cc7-564f8cd89b-n9pvd"]
Aug 04 23:20:51.010850 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:51.010818    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3fe89c-75c5-4385-868d-c5800adc4579" path="/var/lib/kubelet/pods/2e3fe89c-75c5-4385-868d-c5800adc4579/volumes"
Aug 04 23:20:51.011213 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:51.011198    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6f6208-f842-425c-92c8-2f7193b50365" path="/var/lib/kubelet/pods/5c6f6208-f842-425c-92c8-2f7193b50365/volumes"
Aug 04 23:20:55.374108 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:55.374033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:55.435449 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:55.435424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:55.614083 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:55.614027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:20:55.900634 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:20:55.900591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:03.195751 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:03.195717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:21:03.204728 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:03.204692    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:21:08.073984 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:08.073946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:14.780531 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:14.780495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:21:14.781054 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:14.780735    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" podUID="625e245a-1127-428b-aca4-8aa22eafc4d1" containerName="maas-api" containerID="cri-o://0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af" gracePeriod=30
Aug 04 23:21:14.809517 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:14.809495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:14.834919 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:14.834865    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:21:14.835167 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:14.835141    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" podUID="91c4a016-81fe-495a-8a62-b79101ffaf74" containerName="payload-processing" containerID="cri-o://da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b" gracePeriod=30
Aug 04 23:21:15.099218 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.099194    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:21:15.127438 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.127414    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:21:15.161838 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.161817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca\") pod \"625e245a-1127-428b-aca4-8aa22eafc4d1\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") "
Aug 04 23:21:15.161985 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.161868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") pod \"625e245a-1127-428b-aca4-8aa22eafc4d1\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") "
Aug 04 23:21:15.161985 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.161948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrpbc\" (UniqueName: \"kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc\") pod \"625e245a-1127-428b-aca4-8aa22eafc4d1\" (UID: \"625e245a-1127-428b-aca4-8aa22eafc4d1\") "
Aug 04 23:21:15.162173 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.162148    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "625e245a-1127-428b-aca4-8aa22eafc4d1" (UID: "625e245a-1127-428b-aca4-8aa22eafc4d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:21:15.162264 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.162249    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/625e245a-1127-428b-aca4-8aa22eafc4d1-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:15.163999 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.163978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "625e245a-1127-428b-aca4-8aa22eafc4d1" (UID: "625e245a-1127-428b-aca4-8aa22eafc4d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:21:15.164162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.164137    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc" (OuterVolumeSpecName: "kube-api-access-lrpbc") pod "625e245a-1127-428b-aca4-8aa22eafc4d1" (UID: "625e245a-1127-428b-aca4-8aa22eafc4d1"). InnerVolumeSpecName "kube-api-access-lrpbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:21:15.262555 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.262530    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume\") pod \"91c4a016-81fe-495a-8a62-b79101ffaf74\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") "
Aug 04 23:21:15.262658 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.262593    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7dzh\" (UniqueName: \"kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh\") pod \"91c4a016-81fe-495a-8a62-b79101ffaf74\" (UID: \"91c4a016-81fe-495a-8a62-b79101ffaf74\") "
Aug 04 23:21:15.262755 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.262744    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrpbc\" (UniqueName: \"kubernetes.io/projected/625e245a-1127-428b-aca4-8aa22eafc4d1-kube-api-access-lrpbc\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:15.262799 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.262756    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/625e245a-1127-428b-aca4-8aa22eafc4d1-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:15.262919 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.262859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91c4a016-81fe-495a-8a62-b79101ffaf74" (UID: "91c4a016-81fe-495a-8a62-b79101ffaf74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:21:15.265100 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.265079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh" (OuterVolumeSpecName: "kube-api-access-j7dzh") pod "91c4a016-81fe-495a-8a62-b79101ffaf74" (UID: "91c4a016-81fe-495a-8a62-b79101ffaf74"). InnerVolumeSpecName "kube-api-access-j7dzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:21:15.268918 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.268876    2582 generic.go:356] "Generic (PLEG): container finished" podID="91c4a016-81fe-495a-8a62-b79101ffaf74" containerID="da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b" exitCode=0
Aug 04 23:21:15.269001 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.268926    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"
Aug 04 23:21:15.269001 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.268924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" event={"ID":"91c4a016-81fe-495a-8a62-b79101ffaf74","Type":"ContainerDied","Data":"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"}
Aug 04 23:21:15.269001 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.268982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww" event={"ID":"91c4a016-81fe-495a-8a62-b79101ffaf74","Type":"ContainerDied","Data":"9a88c690fc7374c66c3d8ef80e4cc3a9fe30602649defd497f0ddb8e157d0068"}
Aug 04 23:21:15.269126 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.269006    2582 scope.go:117] "RemoveContainer" containerID="da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"
Aug 04 23:21:15.270287 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.270268    2582 generic.go:356] "Generic (PLEG): container finished" podID="625e245a-1127-428b-aca4-8aa22eafc4d1" containerID="0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af" exitCode=0
Aug 04 23:21:15.270370 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.270342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" event={"ID":"625e245a-1127-428b-aca4-8aa22eafc4d1","Type":"ContainerDied","Data":"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"}
Aug 04 23:21:15.270418 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.270367    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"
Aug 04 23:21:15.270418 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.270385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff" event={"ID":"625e245a-1127-428b-aca4-8aa22eafc4d1","Type":"ContainerDied","Data":"4eecdeab791c7310a3e74299334b9c2e2fc841dcd41efaa095e1b92197b2fc28"}
Aug 04 23:21:15.279944 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.279927    2582 scope.go:117] "RemoveContainer" containerID="da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"
Aug 04 23:21:15.280276 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:21:15.280256    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b\": container with ID starting with da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b not found: ID does not exist" containerID="da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"
Aug 04 23:21:15.280354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.280283    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b"} err="failed to get container status \"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b\": rpc error: code = NotFound desc = could not find container \"da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b\": container with ID starting with da7e4c8a789035bd35491d503d047c1ceacbe14a646811112938c049b707b47b not found: ID does not exist"
Aug 04 23:21:15.280354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.280301    2582 scope.go:117] "RemoveContainer" containerID="0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"
Aug 04 23:21:15.289244 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.289223    2582 scope.go:117] "RemoveContainer" containerID="0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"
Aug 04 23:21:15.289505 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:21:15.289489    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af\": container with ID starting with 0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af not found: ID does not exist" containerID="0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"
Aug 04 23:21:15.289549 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.289514    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af"} err="failed to get container status \"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af\": rpc error: code = NotFound desc = could not find container \"0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af\": container with ID starting with 0526f0c5faa03d0983cf838e8abd60651422b679fa6829712792368abe24f4af not found: ID does not exist"
Aug 04 23:21:15.291309 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.291287    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:21:15.293292 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.293263    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1b011-749df98c6-2hpww"]
Aug 04 23:21:15.300969 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.300915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:21:15.308545 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.308525    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d1b011-6c76475cc4-rhsff"]
Aug 04 23:21:15.364004 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.363983    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dzh\" (UniqueName: \"kubernetes.io/projected/91c4a016-81fe-495a-8a62-b79101ffaf74-kube-api-access-j7dzh\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:15.364004 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:15.364004    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c4a016-81fe-495a-8a62-b79101ffaf74-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:16.084064 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:16.083981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:17.011626 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:17.011583    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625e245a-1127-428b-aca4-8aa22eafc4d1" path="/var/lib/kubelet/pods/625e245a-1127-428b-aca4-8aa22eafc4d1/volumes"
Aug 04 23:21:17.012064 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:17.012047    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c4a016-81fe-495a-8a62-b79101ffaf74" path="/var/lib/kubelet/pods/91c4a016-81fe-495a-8a62-b79101ffaf74/volumes"
Aug 04 23:21:20.320608 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:20.320577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:21:20.366528 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:20.366499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:35.340807 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.340738    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" podUID="1dd40264-a810-438e-8cc0-df02308cc8cc" containerName="main" containerID="cri-o://19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18" gracePeriod=45
Aug 04 23:21:35.589669 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.589644    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:21:35.644680 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.644832 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.644832 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.644978 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.644978 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:21:35.644978 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644875    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.644978 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.644919    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb2vv\" (UniqueName: \"kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv\") pod \"1dd40264-a810-438e-8cc0-df02308cc8cc\" (UID: \"1dd40264-a810-438e-8cc0-df02308cc8cc\") "
Aug 04 23:21:35.645240 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.645085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:21:35.645240 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.645117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home" (OuterVolumeSpecName: "home") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:21:35.645354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.645273    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-model-cache\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:35.645354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.645290    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-tmp-dir\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:35.645354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.645302    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-home\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:35.647477 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.647448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:21:35.647725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.647694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv" (OuterVolumeSpecName: "kube-api-access-cb2vv") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "kube-api-access-cb2vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:21:35.647725 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.647716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm" (OuterVolumeSpecName: "dshm") pod "1dd40264-a810-438e-8cc0-df02308cc8cc" (UID: "1dd40264-a810-438e-8cc0-df02308cc8cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 23:21:35.745960 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.745940    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2vv\" (UniqueName: \"kubernetes.io/projected/1dd40264-a810-438e-8cc0-df02308cc8cc-kube-api-access-cb2vv\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:35.745960 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.745960    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd40264-a810-438e-8cc0-df02308cc8cc-tls-certs\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:35.746098 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:35.745969    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd40264-a810-438e-8cc0-df02308cc8cc-dshm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:21:36.379908 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.379858    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dd40264-a810-438e-8cc0-df02308cc8cc" containerID="19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18" exitCode=0
Aug 04 23:21:36.380351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.379928    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"
Aug 04 23:21:36.380351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.379927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" event={"ID":"1dd40264-a810-438e-8cc0-df02308cc8cc","Type":"ContainerDied","Data":"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"}
Aug 04 23:21:36.380351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.379975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp" event={"ID":"1dd40264-a810-438e-8cc0-df02308cc8cc","Type":"ContainerDied","Data":"36d99d4e3b02511daf0a33de73a92849d5b82d1b7f793bfa69bc6d214df6a9bc"}
Aug 04 23:21:36.380351 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.380000    2582 scope.go:117] "RemoveContainer" containerID="19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"
Aug 04 23:21:36.389680 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.389662    2582 scope.go:117] "RemoveContainer" containerID="19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"
Aug 04 23:21:36.389960 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:21:36.389940    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18\": container with ID starting with 19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18 not found: ID does not exist" containerID="19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"
Aug 04 23:21:36.390016 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.389970    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18"} err="failed to get container status \"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18\": rpc error: code = NotFound desc = could not find container \"19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18\": container with ID starting with 19d1b6b88a190c7a0b3488017848cc0f69b964990d41d0173245c47ffdaafb18 not found: ID does not exist"
Aug 04 23:21:36.400049 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.400030    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:21:36.401818 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:36.401799    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d1b011/ipp-route-d1b011-kserve-79bcc9ccc9-tnfxp"]
Aug 04 23:21:37.010861 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:37.010829    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd40264-a810-438e-8cc0-df02308cc8cc" path="/var/lib/kubelet/pods/1dd40264-a810-438e-8cc0-df02308cc8cc/volumes"
Aug 04 23:21:56.801307 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:56.801279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:21:57.373186 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:21:57.373156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:02.867587 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:02.867548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:02.968112 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:02.968083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:03.141359 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:03.141262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:03.369226 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:03.369182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:13.875400 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:13.875368    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:22:13.875907 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:13.875581    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" podUID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" containerName="maas-api" containerID="cri-o://645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f" gracePeriod=30
Aug 04 23:22:13.899913 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:13.899856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:13.964832 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:13.964802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:14.019449 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.019418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:14.117774 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.117749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:14.118373 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.118355    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:22:14.304158 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.304073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") pod \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") "
Aug 04 23:22:14.304158 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.304109    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbjv5\" (UniqueName: \"kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5\") pod \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") "
Aug 04 23:22:14.304361 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.304172    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca\") pod \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\" (UID: \"028dd80e-7aa3-4116-80d1-a8366e2b17ee\") "
Aug 04 23:22:14.304613 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.304584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "028dd80e-7aa3-4116-80d1-a8366e2b17ee" (UID: "028dd80e-7aa3-4116-80d1-a8366e2b17ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:22:14.306306 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.306284    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "028dd80e-7aa3-4116-80d1-a8366e2b17ee" (UID: "028dd80e-7aa3-4116-80d1-a8366e2b17ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:22:14.306399 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.306349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5" (OuterVolumeSpecName: "kube-api-access-rbjv5") pod "028dd80e-7aa3-4116-80d1-a8366e2b17ee" (UID: "028dd80e-7aa3-4116-80d1-a8366e2b17ee"). InnerVolumeSpecName "kube-api-access-rbjv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:22:14.405798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.405764    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028dd80e-7aa3-4116-80d1-a8366e2b17ee-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:14.405798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.405793    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028dd80e-7aa3-4116-80d1-a8366e2b17ee-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:14.405798 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.405803    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjv5\" (UniqueName: \"kubernetes.io/projected/028dd80e-7aa3-4116-80d1-a8366e2b17ee-kube-api-access-rbjv5\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:14.440566 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.440534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:14.573474 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.573397    2582 generic.go:356] "Generic (PLEG): container finished" podID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" containerID="645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f" exitCode=0
Aug 04 23:22:14.573474 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.573453    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"
Aug 04 23:22:14.573637 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.573477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" event={"ID":"028dd80e-7aa3-4116-80d1-a8366e2b17ee","Type":"ContainerDied","Data":"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"}
Aug 04 23:22:14.573637 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.573550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd" event={"ID":"028dd80e-7aa3-4116-80d1-a8366e2b17ee","Type":"ContainerDied","Data":"b4d8509c8dac5e29e888c8e582c745de98f8c9e49df851f5a48e406373f42690"}
Aug 04 23:22:14.573637 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.573575    2582 scope.go:117] "RemoveContainer" containerID="645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"
Aug 04 23:22:14.582691 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.582674    2582 scope.go:117] "RemoveContainer" containerID="645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"
Aug 04 23:22:14.582972 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:22:14.582950    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f\": container with ID starting with 645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f not found: ID does not exist" containerID="645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"
Aug 04 23:22:14.583057 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.582979    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f"} err="failed to get container status \"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f\": rpc error: code = NotFound desc = could not find container \"645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f\": container with ID starting with 645ee50381c825bd3703ea78cd1b879f79288700c9731a7d53464fbd00214c3f not found: ID does not exist"
Aug 04 23:22:14.593183 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.593159    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:22:14.595214 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:14.595194    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edf378-589b4bfb67-s7nwd"]
Aug 04 23:22:15.010395 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:15.010361    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" path="/var/lib/kubelet/pods/028dd80e-7aa3-4116-80d1-a8366e2b17ee/volumes"
Aug 04 23:22:20.308178 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:20.308131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:20.362334 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:20.362294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:20.567783 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:20.567703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:20.807135 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:20.807091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:31.309330 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.309248    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:22:31.309914 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.309489    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerName="maas-api" containerID="cri-o://7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca" gracePeriod=30
Aug 04 23:22:31.342096 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.342063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:31.364861 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.364827    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:22:31.365165 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.365135    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerName="payload-processing" containerID="cri-o://dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461" gracePeriod=30
Aug 04 23:22:31.373175 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.373143    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:22:31.373465 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.373382    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" containerName="payload-pre-processing" containerID="cri-o://0c69c23563c40da6dd310d42bda662f6ec2acfba962d01c4044e584a5b453bba" gracePeriod=30
Aug 04 23:22:31.440452 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.440422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:31.498784 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.498749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:31.503761 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.503714    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.70:9004: connect: connection refused"
Aug 04 23:22:31.503761 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.503749    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.68:8443/health\": dial tcp 10.134.0.68:8443: connect: connection refused"
Aug 04 23:22:31.503988 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.503720    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.69:9004: connect: connection refused"
Aug 04 23:22:31.649878 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.649849    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:22:31.653649 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.653626    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:22:31.664497 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.664467    2582 generic.go:356] "Generic (PLEG): container finished" podID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerID="7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca" exitCode=0
Aug 04 23:22:31.664682 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.664558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" event={"ID":"f4f89345-6a74-43a6-97e1-24854a4b3f25","Type":"ContainerDied","Data":"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"}
Aug 04 23:22:31.664682 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.664612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x" event={"ID":"f4f89345-6a74-43a6-97e1-24854a4b3f25","Type":"ContainerDied","Data":"2cb92766c4973243d2febf422333b4c73b548eb1c74fb2c5ccaf3da13e75a027"}
Aug 04 23:22:31.664682 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.664645    2582 scope.go:117] "RemoveContainer" containerID="7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"
Aug 04 23:22:31.664855 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.664783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"
Aug 04 23:22:31.667783 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.667761    2582 generic.go:356] "Generic (PLEG): container finished" podID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerID="dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461" exitCode=0
Aug 04 23:22:31.668059 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.668045    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"
Aug 04 23:22:31.668439 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.668418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" event={"ID":"8a2bd1eb-2fda-410e-94ad-498279bf6dce","Type":"ContainerDied","Data":"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"}
Aug 04 23:22:31.668591 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.668575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s" event={"ID":"8a2bd1eb-2fda-410e-94ad-498279bf6dce","Type":"ContainerDied","Data":"0eaa203c45a7ed8b5f13aa75ec1e10f945b7764a7d3a75c02a5bef0ea3880fdb"}
Aug 04 23:22:31.669692 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.669671    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ab24551-fff8-426a-832b-c000aebf2eea" containerID="0c69c23563c40da6dd310d42bda662f6ec2acfba962d01c4044e584a5b453bba" exitCode=0
Aug 04 23:22:31.669930 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.669913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" event={"ID":"7ab24551-fff8-426a-832b-c000aebf2eea","Type":"ContainerDied","Data":"0c69c23563c40da6dd310d42bda662f6ec2acfba962d01c4044e584a5b453bba"}
Aug 04 23:22:31.679503 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.679433    2582 scope.go:117] "RemoveContainer" containerID="7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"
Aug 04 23:22:31.683427 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:22:31.683289    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca\": container with ID starting with 7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca not found: ID does not exist" containerID="7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"
Aug 04 23:22:31.683427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.683327    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca"} err="failed to get container status \"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca\": rpc error: code = NotFound desc = could not find container \"7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca\": container with ID starting with 7f653443dc6b7ca78c8577fa21fe2f09d00fa0eabb0a8cb07aa15ccdc7f20cca not found: ID does not exist"
Aug 04 23:22:31.683427 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.683358    2582 scope.go:117] "RemoveContainer" containerID="dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"
Aug 04 23:22:31.706026 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.706011    2582 scope.go:117] "RemoveContainer" containerID="dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"
Aug 04 23:22:31.706281 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:22:31.706258    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461\": container with ID starting with dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461 not found: ID does not exist" containerID="dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"
Aug 04 23:22:31.706332 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.706288    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461"} err="failed to get container status \"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461\": rpc error: code = NotFound desc = could not find container \"dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461\": container with ID starting with dd01883ae00d5d6d224057c14217fa2329f3455a2301637a67df2be29a9da461 not found: ID does not exist"
Aug 04 23:22:31.707939 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.707905    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:22:31.752699 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.752673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca\") pod \"f4f89345-6a74-43a6-97e1-24854a4b3f25\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") "
Aug 04 23:22:31.752817 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.752713    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzkwf\" (UniqueName: \"kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf\") pod \"7ab24551-fff8-426a-832b-c000aebf2eea\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") "
Aug 04 23:22:31.752817 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.752729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume\") pod \"7ab24551-fff8-426a-832b-c000aebf2eea\" (UID: \"7ab24551-fff8-426a-832b-c000aebf2eea\") "
Aug 04 23:22:31.752963 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.752947    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls\") pod \"f4f89345-6a74-43a6-97e1-24854a4b3f25\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") "
Aug 04 23:22:31.753013 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.752999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmnm\" (UniqueName: \"kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm\") pod \"f4f89345-6a74-43a6-97e1-24854a4b3f25\" (UID: \"f4f89345-6a74-43a6-97e1-24854a4b3f25\") "
Aug 04 23:22:31.753064 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn5q8\" (UniqueName: \"kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8\") pod \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") "
Aug 04 23:22:31.753113 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume\") pod \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\" (UID: \"8a2bd1eb-2fda-410e-94ad-498279bf6dce\") "
Aug 04 23:22:31.753113 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4f89345-6a74-43a6-97e1-24854a4b3f25" (UID: "f4f89345-6a74-43a6-97e1-24854a4b3f25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:22:31.753223 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753126    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ab24551-fff8-426a-832b-c000aebf2eea" (UID: "7ab24551-fff8-426a-832b-c000aebf2eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:22:31.753394 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753374    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4f89345-6a74-43a6-97e1-24854a4b3f25-openshift-service-ca\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.753497 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753396    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab24551-fff8-426a-832b-c000aebf2eea-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.753859 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.753812    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a2bd1eb-2fda-410e-94ad-498279bf6dce" (UID: "8a2bd1eb-2fda-410e-94ad-498279bf6dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:22:31.755221 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.755200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf" (OuterVolumeSpecName: "kube-api-access-vzkwf") pod "7ab24551-fff8-426a-832b-c000aebf2eea" (UID: "7ab24551-fff8-426a-832b-c000aebf2eea"). InnerVolumeSpecName "kube-api-access-vzkwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:22:31.755316 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.755254    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm" (OuterVolumeSpecName: "kube-api-access-lqmnm") pod "f4f89345-6a74-43a6-97e1-24854a4b3f25" (UID: "f4f89345-6a74-43a6-97e1-24854a4b3f25"). InnerVolumeSpecName "kube-api-access-lqmnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:22:31.755316 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.755282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4f89345-6a74-43a6-97e1-24854a4b3f25" (UID: "f4f89345-6a74-43a6-97e1-24854a4b3f25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 23:22:31.755407 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.755336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8" (OuterVolumeSpecName: "kube-api-access-bn5q8") pod "8a2bd1eb-2fda-410e-94ad-498279bf6dce" (UID: "8a2bd1eb-2fda-410e-94ad-498279bf6dce"). InnerVolumeSpecName "kube-api-access-bn5q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:22:31.854315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.854234    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzkwf\" (UniqueName: \"kubernetes.io/projected/7ab24551-fff8-426a-832b-c000aebf2eea-kube-api-access-vzkwf\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.854315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.854268    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4f89345-6a74-43a6-97e1-24854a4b3f25-maas-api-tls\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.854315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.854278    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmnm\" (UniqueName: \"kubernetes.io/projected/f4f89345-6a74-43a6-97e1-24854a4b3f25-kube-api-access-lqmnm\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.854315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.854288    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5q8\" (UniqueName: \"kubernetes.io/projected/8a2bd1eb-2fda-410e-94ad-498279bf6dce-kube-api-access-bn5q8\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.854315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.854297    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2bd1eb-2fda-410e-94ad-498279bf6dce-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:31.877312 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.877282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:31.988056 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.988032    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:22:31.991433 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:31.991412    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec06a8-ddf5d9cbb-cn57x"]
Aug 04 23:22:32.002370 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.002345    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:22:32.005690 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.005668    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ec06a8-74db86fbc4-tdr2s"]
Aug 04 23:22:32.678325 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.678292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9" event={"ID":"7ab24551-fff8-426a-832b-c000aebf2eea","Type":"ContainerDied","Data":"7f7873a1e1213d89560f6a7286ff048eb49d8345931c714ac9fe06cfe0e0a8af"}
Aug 04 23:22:32.678729 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.678349    2582 scope.go:117] "RemoveContainer" containerID="0c69c23563c40da6dd310d42bda662f6ec2acfba962d01c4044e584a5b453bba"
Aug 04 23:22:32.678729 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.678306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"
Aug 04 23:22:32.699177 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.699148    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:22:32.705654 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:32.705630    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ec06a8-dc445f8c6-lwnb9"]
Aug 04 23:22:33.011495 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:33.011422    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" path="/var/lib/kubelet/pods/7ab24551-fff8-426a-832b-c000aebf2eea/volumes"
Aug 04 23:22:33.011830 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:33.011805    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" path="/var/lib/kubelet/pods/8a2bd1eb-2fda-410e-94ad-498279bf6dce/volumes"
Aug 04 23:22:33.012250 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:33.012234    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" path="/var/lib/kubelet/pods/f4f89345-6a74-43a6-97e1-24854a4b3f25/volumes"
Aug 04 23:22:37.606162 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:37.606119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:37.706380 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:37.706348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:37.869298 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:37.869265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:38.102710 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:38.102669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:48.725665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.725626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:48.757494 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.757462    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:22:48.757738 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.757702    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" podUID="64fbf540-8da9-45af-a200-1f6393bf9c63" containerName="payload-pre-processing" containerID="cri-o://1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb" gracePeriod=30
Aug 04 23:22:48.789115 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.789084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:48.884142 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.884103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:48.933957 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:48.933920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:49.005443 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.005420    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:22:49.104315 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.104284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzn6d\" (UniqueName: \"kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d\") pod \"64fbf540-8da9-45af-a200-1f6393bf9c63\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") "
Aug 04 23:22:49.104479 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.104373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume\") pod \"64fbf540-8da9-45af-a200-1f6393bf9c63\" (UID: \"64fbf540-8da9-45af-a200-1f6393bf9c63\") "
Aug 04 23:22:49.104708 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.104687    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64fbf540-8da9-45af-a200-1f6393bf9c63" (UID: "64fbf540-8da9-45af-a200-1f6393bf9c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 23:22:49.106542 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.106517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d" (OuterVolumeSpecName: "kube-api-access-nzn6d") pod "64fbf540-8da9-45af-a200-1f6393bf9c63" (UID: "64fbf540-8da9-45af-a200-1f6393bf9c63"). InnerVolumeSpecName "kube-api-access-nzn6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 23:22:49.205638 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.205612    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbf540-8da9-45af-a200-1f6393bf9c63-plugins-config-volume\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:49.205638 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.205636    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzn6d\" (UniqueName: \"kubernetes.io/projected/64fbf540-8da9-45af-a200-1f6393bf9c63-kube-api-access-nzn6d\") on node \"ip-10-0-142-146.ec2.internal\" DevicePath \"\""
Aug 04 23:22:49.244829 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.244770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:49.764464 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.764428    2582 generic.go:356] "Generic (PLEG): container finished" podID="64fbf540-8da9-45af-a200-1f6393bf9c63" containerID="1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb" exitCode=0
Aug 04 23:22:49.764887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.764473    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"
Aug 04 23:22:49.764887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.764508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" event={"ID":"64fbf540-8da9-45af-a200-1f6393bf9c63","Type":"ContainerDied","Data":"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"}
Aug 04 23:22:49.764887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.764561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9" event={"ID":"64fbf540-8da9-45af-a200-1f6393bf9c63","Type":"ContainerDied","Data":"61fc680def433f2d7bd5f2fc53693d716875a8062af7909d1947e7e986374267"}
Aug 04 23:22:49.764887 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.764583    2582 scope.go:117] "RemoveContainer" containerID="1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"
Aug 04 23:22:49.773456 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.773433    2582 scope.go:117] "RemoveContainer" containerID="1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"
Aug 04 23:22:49.773704 ip-10-0-142-146 kubenswrapper[2582]: E0804 23:22:49.773685    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb\": container with ID starting with 1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb not found: ID does not exist" containerID="1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"
Aug 04 23:22:49.773755 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.773712    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb"} err="failed to get container status \"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb\": rpc error: code = NotFound desc = could not find container \"1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb\": container with ID starting with 1a9cdbc3fe4e4111ce2fa4bf6914a57e2e7ac3026019534dbea943861b89cecb not found: ID does not exist"
Aug 04 23:22:49.785077 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.785056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:22:49.788723 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:49.788702    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-55909a-7d45fbfc8d-mjtx9"]
Aug 04 23:22:51.010765 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:51.010731    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fbf540-8da9-45af-a200-1f6393bf9c63" path="/var/lib/kubelet/pods/64fbf540-8da9-45af-a200-1f6393bf9c63/volumes"
Aug 04 23:22:54.996043 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:54.996015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:55.052095 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:55.052061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:55.257443 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:55.257358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:55.478448 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:55.478402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:22:55.878468 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:22:55.878397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d5c994c8-4gtrk_248b1e83-2492-4ef8-86ab-ac2ecabe2922/authorino/0.log"
Aug 04 23:23:01.625012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:01.624982    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb_c9b6ea31-6cb3-443e-98f3-3123dc899aec/copy-manifests/0.log"
Aug 04 23:23:01.637834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:01.637813    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb_c9b6ea31-6cb3-443e-98f3-3123dc899aec/manager/0.log"
Aug 04 23:23:02.214189 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:02.214154    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cf454fbc-j2w5m_0b1ee76e-a649-4319-aece-b602bb8795a1/maas-api/0.log"
Aug 04 23:23:03.432985 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:03.432951    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d5c994c8-4gtrk_248b1e83-2492-4ef8-86ab-ac2ecabe2922/authorino/0.log"
Aug 04 23:23:04.698851 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:04.698816    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c576658-pjsgv_3fdb8d18-834d-4245-b7d0-740a947ff360/kube-auth-proxy/0.log"
Aug 04 23:23:05.033882 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:05.033810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-6lgcs_2828a025-f588-48df-b1eb-a5eb08ea6304/payload-pre-processing/0.log"
Aug 04 23:23:05.151978 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:05.151947    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:23:05.266187 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:05.266164    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67655c4d4d-kzn5x_a520a83c-b775-4e8d-8236-f85692310229/router/0.log"
Aug 04 23:23:05.615165 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:05.615137    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zwhhd_a6ebf54a-d123-4bad-be45-7c3f704f940b/main/0.log"
Aug 04 23:23:05.851959 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:05.851915    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jhj6n_df467462-3f0a-4be4-bde3-83baf29a5380/main/0.log"
Aug 04 23:23:06.079239 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:06.079213    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2f6zw_5edac72f-db15-4507-9758-60f66b0ed7e1/main/0.log"
Aug 04 23:23:11.483580 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:11.483539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:23:11.578760 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:11.578725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:23:14.117936 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:14.117879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5855j_05b4f6e1-6a4e-4e9b-9415-5e86cbca6167/global-pull-secret-syncer/0.log"
Aug 04 23:23:14.238327 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:14.238302    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log"
Aug 04 23:23:14.386665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:14.386586    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-146.ec2.internal_b56d4e9c7e28a671fb29fb60ed5ddf2b/haproxy/0.log"
Aug 04 23:23:19.307345 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:19.307306    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d5c994c8-4gtrk_248b1e83-2492-4ef8-86ab-ac2ecabe2922/authorino/0.log"
Aug 04 23:23:20.204812 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:20.204776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ch8lh_c6662a4b-5ab9-4de4-acc5-8d8cf095e13c/konnectivity-agent/0.log" limit=1024
Aug 04 23:23:21.520192 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.520116    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-state-metrics/0.log"
Aug 04 23:23:21.538007 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.537983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-rbac-proxy-main/0.log"
Aug 04 23:23:21.557817 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.557795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-rsfct_fce7924e-ce59-4806-850f-80cc8d8c00b3/kube-rbac-proxy-self/0.log"
Aug 04 23:23:21.609875 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.609851    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-6rgg2_da3b55d0-6766-4865-afeb-7241dd4a5e88/monitoring-plugin/0.log"
Aug 04 23:23:21.716391 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.716367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/node-exporter/0.log"
Aug 04 23:23:21.745795 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.745763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/kube-rbac-proxy/0.log"
Aug 04 23:23:21.767773 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.767745    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bp984_895593f6-4d7d-44d3-acbe-a67910294dcc/init-textfile/0.log"
Aug 04 23:23:21.868698 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.868670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/kube-rbac-proxy-main/0.log"
Aug 04 23:23:21.889549 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.889526    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/kube-rbac-proxy-self/0.log"
Aug 04 23:23:21.910727 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.910707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-twb9m_fae58b47-ec22-4e75-9142-0c59114a3b9e/openshift-state-metrics/0.log"
Aug 04 23:23:21.955286 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.955268    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/prometheus/0.log"
Aug 04 23:23:21.981922 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:21.981889    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/config-reloader/0.log"
Aug 04 23:23:22.012834 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.012814    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/thanos-sidecar/0.log"
Aug 04 23:23:22.032654 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.032626    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/kube-rbac-proxy-web/0.log"
Aug 04 23:23:22.053354 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.053329    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/kube-rbac-proxy/0.log"
Aug 04 23:23:22.074012 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.073989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/kube-rbac-proxy-thanos/0.log"
Aug 04 23:23:22.103815 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.103791    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bddc87b2-a2f2-427f-b828-ec2a1c9abfb4/init-config-reloader/0.log"
Aug 04 23:23:22.304749 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.304653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/thanos-query/0.log"
Aug 04 23:23:22.323872 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.323851    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-web/0.log"
Aug 04 23:23:22.345500 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.345465    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy/0.log"
Aug 04 23:23:22.370363 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.370341    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/prom-label-proxy/0.log"
Aug 04 23:23:22.389881 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.389855    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-rules/0.log"
Aug 04 23:23:22.410043 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:22.410022    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578447c689-j4kn8_48499a1b-4d1a-453b-a4bb-a5360294d0f2/kube-rbac-proxy-metrics/0.log"
Aug 04 23:23:24.209761 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.209731    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"]
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210091    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c4a016-81fe-495a-8a62-b79101ffaf74" containerName="payload-processing"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210102    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c4a016-81fe-495a-8a62-b79101ffaf74" containerName="payload-processing"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210114    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerName="maas-api"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210120    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerName="maas-api"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210127    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerName="payload-processing"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210132    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerName="payload-processing"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210142    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6f6208-f842-425c-92c8-2f7193b50365" containerName="payload-processing"
Aug 04 23:23:24.210152 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210147    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6f6208-f842-425c-92c8-2f7193b50365" containerName="payload-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210161    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210166    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210174    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210179    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210187    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fbf540-8da9-45af-a200-1f6393bf9c63" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210193    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fbf540-8da9-45af-a200-1f6393bf9c63" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210204    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625e245a-1127-428b-aca4-8aa22eafc4d1" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210209    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="625e245a-1127-428b-aca4-8aa22eafc4d1" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210215    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd40264-a810-438e-8cc0-df02308cc8cc" containerName="main"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd40264-a810-438e-8cc0-df02308cc8cc" containerName="main"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210227    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3fe89c-75c5-4385-868d-c5800adc4579" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210232    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3fe89c-75c5-4385-868d-c5800adc4579" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210284    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f89345-6a74-43a6-97e1-24854a4b3f25" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210290    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c4a016-81fe-495a-8a62-b79101ffaf74" containerName="payload-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210296    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd40264-a810-438e-8cc0-df02308cc8cc" containerName="main"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210302    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab24551-fff8-426a-832b-c000aebf2eea" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210307    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2bd1eb-2fda-410e-94ad-498279bf6dce" containerName="payload-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210312    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="028dd80e-7aa3-4116-80d1-a8366e2b17ee" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210317    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3fe89c-75c5-4385-868d-c5800adc4579" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210325    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fbf540-8da9-45af-a200-1f6393bf9c63" containerName="payload-pre-processing"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210330    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="625e245a-1127-428b-aca4-8aa22eafc4d1" containerName="maas-api"
Aug 04 23:23:24.210412 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.210336    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6f6208-f842-425c-92c8-2f7193b50365" containerName="payload-processing"
Aug 04 23:23:24.213418 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.213401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.215287 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.215271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6vkp5\"/\"openshift-service-ca.crt\""
Aug 04 23:23:24.215440 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.215391    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6vkp5\"/\"kube-root-ca.crt\""
Aug 04 23:23:24.216211 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.216194    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6vkp5\"/\"default-dockercfg-grvmj\""
Aug 04 23:23:24.222804 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.222781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"]
Aug 04 23:23:24.312665 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.312632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pccbm\" (UniqueName: \"kubernetes.io/projected/154b0722-02de-4bf7-83a5-275d85463b2a-kube-api-access-pccbm\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.312805 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.312689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-podres\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.312805 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.312722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-sys\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.312805 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.312742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-proc\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.312805 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.312758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-lib-modules\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.368966 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.368889    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/2.log"
Aug 04 23:23:24.372597 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.372582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wzprd_14a577bd-ca9a-4993-b06c-b5013929952f/console-operator/3.log"
Aug 04 23:23:24.413305 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pccbm\" (UniqueName: \"kubernetes.io/projected/154b0722-02de-4bf7-83a5-275d85463b2a-kube-api-access-pccbm\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413305 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-podres\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-sys\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-proc\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-lib-modules\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-sys\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-proc\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-podres\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.413481 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.413470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/154b0722-02de-4bf7-83a5-275d85463b2a-lib-modules\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.421114 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.421090    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pccbm\" (UniqueName: \"kubernetes.io/projected/154b0722-02de-4bf7-83a5-275d85463b2a-kube-api-access-pccbm\") pod \"perf-node-gather-daemonset-d9c4b\" (UID: \"154b0722-02de-4bf7-83a5-275d85463b2a\") " pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.523698 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.523630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.657857 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.657825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"]
Aug 04 23:23:24.660515 ip-10-0-142-146 kubenswrapper[2582]: W0804 23:23:24.660487    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod154b0722_02de_4bf7_83a5_275d85463b2a.slice/crio-8e798c38bb29a9a77eaa07891128d8e80c8dab1b8913f2673722f4e79ddc442a WatchSource:0}: Error finding container 8e798c38bb29a9a77eaa07891128d8e80c8dab1b8913f2673722f4e79ddc442a: Status 404 returned error can't find the container with id 8e798c38bb29a9a77eaa07891128d8e80c8dab1b8913f2673722f4e79ddc442a
Aug 04 23:23:24.932594 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.932557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b" event={"ID":"154b0722-02de-4bf7-83a5-275d85463b2a","Type":"ContainerStarted","Data":"c3b150db380a520686607ff20c4f07091a7b2e21f9deac4a07a31642e4634d40"}
Aug 04 23:23:24.932774 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.932620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b" event={"ID":"154b0722-02de-4bf7-83a5-275d85463b2a","Type":"ContainerStarted","Data":"8e798c38bb29a9a77eaa07891128d8e80c8dab1b8913f2673722f4e79ddc442a"}
Aug 04 23:23:24.932774 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.932652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:24.946156 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:24.946083    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b" podStartSLOduration=0.946063774 podStartE2EDuration="946.063774ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 23:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 23:23:24.945565032 +0000 UTC m=+3742.487058231" watchObservedRunningTime="2026-08-04 23:23:24.946063774 +0000 UTC m=+3742.487556948"
Aug 04 23:23:26.806880 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:26.806850    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fggh6_e68ddc17-b853-463e-9a50-720ea2758229/dns/0.log"
Aug 04 23:23:26.827101 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:26.827076    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fggh6_e68ddc17-b853-463e-9a50-720ea2758229/kube-rbac-proxy/0.log"
Aug 04 23:23:26.938373 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:26.938346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsjtt_373d37ba-f50e-4fc6-bf05-56f1c09dce5f/dns-node-resolver/0.log"
Aug 04 23:23:27.490541 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:27.490508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4z68c_0321c2f7-349d-41ee-844b-7710cdab0e8c/node-ca/0.log"
Aug 04 23:23:28.780884 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:28.780846    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c576658-pjsgv_3fdb8d18-834d-4245-b7d0-740a947ff360/kube-auth-proxy/0.log"
Aug 04 23:23:28.876416 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:28.876391    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-6lgcs_2828a025-f588-48df-b1eb-a5eb08ea6304/payload-pre-processing/0.log"
Aug 04 23:23:28.903443 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:28.903418    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:23:28.923925 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:28.923880    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67655c4d4d-kzn5x_a520a83c-b775-4e8d-8236-f85692310229/router/0.log"
Aug 04 23:23:29.742185 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:29.742153    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqhwj_8fbf8a0c-ad84-445a-beab-7a2f37afd2a7/serve-healthcheck-canary/0.log"
Aug 04 23:23:30.257291 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.257259    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-fj4tc_eb8717d6-9bab-47ff-be87-16266aa3e79d/insights-operator/0.log"
Aug 04 23:23:30.259616 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.259586    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-fj4tc_eb8717d6-9bab-47ff-be87-16266aa3e79d/insights-operator/1.log"
Aug 04 23:23:30.346129 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.346109    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngflx_0724525b-ad26-4e95-91f3-4687479a9a5a/kube-rbac-proxy/0.log"
Aug 04 23:23:30.365957 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.365936    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngflx_0724525b-ad26-4e95-91f3-4687479a9a5a/exporter/0.log"
Aug 04 23:23:30.387155 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.387135    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngflx_0724525b-ad26-4e95-91f3-4687479a9a5a/extractor/0.log"
Aug 04 23:23:30.947495 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:30.947470    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6vkp5/perf-node-gather-daemonset-d9c4b"
Aug 04 23:23:33.396400 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:33.396340    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb_c9b6ea31-6cb3-443e-98f3-3123dc899aec/manager/0.log"
Aug 04 23:23:33.415703 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:33.415677    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6cdbbfdb-2fsxb_c9b6ea31-6cb3-443e-98f3-3123dc899aec/copy-manifests/0.log"
Aug 04 23:23:34.894815 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:34.894789    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76f49b9fdf-rxcsm_eefd7f8a-8b34-4484-ba58-54c1dcbff28a/manager/0.log"
Aug 04 23:23:35.562059 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:35.562032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c576658-pjsgv_3fdb8d18-834d-4245-b7d0-740a947ff360/kube-auth-proxy/0.log"
Aug 04 23:23:35.637709 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:35.637685    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-6lgcs_2828a025-f588-48df-b1eb-a5eb08ea6304/payload-pre-processing/0.log"
Aug 04 23:23:35.660956 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:35.660929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:23:35.674015 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:35.673989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67655c4d4d-kzn5x_a520a83c-b775-4e8d-8236-f85692310229/router/0.log"
Aug 04 23:23:37.364502 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:37.364473    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c576658-pjsgv_3fdb8d18-834d-4245-b7d0-740a947ff360/kube-auth-proxy/0.log"
Aug 04 23:23:37.435403 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:37.435366    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-6lgcs_2828a025-f588-48df-b1eb-a5eb08ea6304/payload-pre-processing/0.log"
Aug 04 23:23:37.458198 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:37.458173    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x7mv_ee63be74-bd8e-4d9e-8315-0706aeeb283e/payload-processing/0.log"
Aug 04 23:23:37.474666 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:37.474644    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67655c4d4d-kzn5x_a520a83c-b775-4e8d-8236-f85692310229/router/0.log"
Aug 04 23:23:40.282063 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:40.281990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-8jc8k_56f5848b-6584-4442-b731-8d93865c3a11/kube-storage-version-migrator-operator/1.log"
Aug 04 23:23:40.283099 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:40.283072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-8jc8k_56f5848b-6584-4442-b731-8d93865c3a11/kube-storage-version-migrator-operator/0.log"
Aug 04 23:23:41.343945 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.343912    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/kube-multus-additional-cni-plugins/0.log"
Aug 04 23:23:41.362646 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.362623    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/egress-router-binary-copy/0.log"
Aug 04 23:23:41.383860 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.383839    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/cni-plugins/0.log"
Aug 04 23:23:41.403196 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.403178    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/bond-cni-plugin/0.log"
Aug 04 23:23:41.422156 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.422134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/routeoverride-cni/0.log"
Aug 04 23:23:41.441022 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.441000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/whereabouts-cni-bincopy/0.log"
Aug 04 23:23:41.460102 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.460081    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89svx_a38d7069-0118-40fd-ba16-aaffbb74a00c/whereabouts-cni/0.log"
Aug 04 23:23:41.677166 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.677139    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-stpwt_a449619b-0633-4b78-9196-cd078b370979/kube-multus/0.log"
Aug 04 23:23:41.781196 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.781169    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bqcgb_03c5ee73-fd6c-4f21-a662-ec218702b7d9/network-metrics-daemon/0.log"
Aug 04 23:23:41.798686 ip-10-0-142-146 kubenswrapper[2582]: I0804 23:23:41.798663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bqcgb_03c5ee73-fd6c-4f21-a662-ec218702b7d9/kube-rbac-proxy/0.log"