Jul 11 03:42:40.363378 ip-10-0-138-240 systemd[1]: Starting Kubernetes Kubelet... Jul 11 03:42:40.817346 ip-10-0-138-240 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 03:42:40.817346 ip-10-0-138-240 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 11 03:42:40.817346 ip-10-0-138-240 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 03:42:40.817346 ip-10-0-138-240 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 11 03:42:40.817346 ip-10-0-138-240 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 03:42:40.819314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.819228 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 11 03:42:40.824218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824198 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.824218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824217 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824221 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824225 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824228 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824231 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824234 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824237 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824240 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824243 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824247 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824249 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824253 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824255 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824258 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824261 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824264 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824266 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824269 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824271 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.824285 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824274 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824277 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824280 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824282 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824292 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824295 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824298 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824301 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824303 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824306 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824309 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824311 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824314 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824317 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824319 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824321 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824324 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824327 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824329 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824332 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.824734 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824334 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824344 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824347 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824350 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824352 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824355 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824357 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824360 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824362 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824365 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824367 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824372 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824375 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824378 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824382 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824384 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824387 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824390 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824392 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824394 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.825218 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824398 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824402 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824406 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824408 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824411 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824414 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824417 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824420 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824422 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824425 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824427 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824430 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824433 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824437 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824440 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824443 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824448 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824451 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824454 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.825728 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824456 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824459 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824463 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824466 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824469 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824471 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824474 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824478 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824480 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824483 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824486 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824488 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824491 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824494 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824496 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824499 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824501 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824504 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824506 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.826193 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.824509 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825254 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825260 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825263 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825266 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825269 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825272 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825275 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825278 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825282 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825284 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825287 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825290 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825293 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825295 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825298 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825301 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825303 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825306 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825309 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.826696 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825311 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825314 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825317 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825319 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825322 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825325 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825328 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825330 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825333 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825336 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825339 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825343 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825346 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825349 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825351 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825354 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825356 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825359 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825362 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.827194 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825365 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825368 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825370 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825375 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825378 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825381 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825385 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825387 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825390 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825393 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825396 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825400 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825403 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825405 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825408 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825411 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825413 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825415 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825418 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.827686 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825420 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825423 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825425 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825428 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825430 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825433 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825435 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825438 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825440 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825443 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825446 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825449 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825451 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825454 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825456 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825459 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825461 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825464 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825467 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825470 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.828152 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825472 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825476 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825478 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825481 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825483 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825486 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825488 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825490 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825493 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825496 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825498 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825501 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825503 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825506 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825508 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825511 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825513 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825516 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825518 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.828628 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825521 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825523 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825526 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825924 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825929 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825932 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825935 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825938 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825940 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825943 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825948 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825954 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825958 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825961 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825964 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825967 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825969 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825972 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825974 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.829102 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825977 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825980 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825982 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825984 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825987 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825989 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825992 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825994 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.825999 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826002 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826005 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826007 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826010 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826012 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826016 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826018 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826022 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826025 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826027 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.829563 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826030 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826048 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826051 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826054 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826056 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826070 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826074 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826077 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826080 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826082 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826085 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826087 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826091 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826093 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826095 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826098 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826100 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826103 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826105 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826108 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.830020 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826111 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826113 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826116 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826119 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826121 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826123 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826126 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826128 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826130 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826135 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826138 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826141 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826144 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826147 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826149 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826152 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826154 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826157 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826163 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826165 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.830519 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826168 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826171 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826173 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826176 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826179 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826181 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826184 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826186 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826189 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826191 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826194 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826197 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826199 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826202 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826205 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826207 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826210 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826212 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826215 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826217 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.831004 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826220 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826224 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826227 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826229 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826316 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826326 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826343 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826350 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826355 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826358 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826363 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826368 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826372 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826375 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826378 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826381 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826384 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826387 2565 flags.go:64] FLAG: --cgroup-root="" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826390 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826393 2565 flags.go:64] FLAG: --client-ca-file="" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826396 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826399 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826403 2565 flags.go:64] FLAG: --cluster-domain="" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826406 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 11 03:42:40.831490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826409 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826412 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826416 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826420 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826423 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826426 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826429 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826432 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826435 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826440 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826443 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826446 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826450 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826453 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826456 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826459 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826462 2565 flags.go:64] FLAG: --enable-server="true" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826465 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826469 2565 flags.go:64] FLAG: --event-burst="100" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826473 2565 flags.go:64] FLAG: --event-qps="50" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826477 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826480 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826483 2565 flags.go:64] FLAG: --eviction-hard="" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826486 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 11 03:42:40.832071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826489 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826492 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826495 2565 flags.go:64] FLAG: --eviction-soft="" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826498 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826501 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826504 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826507 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826510 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826512 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826515 2565 flags.go:64] FLAG: --feature-gates="" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826519 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826522 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826525 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826528 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826531 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826534 2565 flags.go:64] FLAG: --help="false" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826537 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-138-240.ec2.internal" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826542 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826544 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826547 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826551 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826554 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826557 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 11 03:42:40.832729 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826559 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826562 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826565 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826568 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826571 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826574 2565 flags.go:64] FLAG: --kube-reserved="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826578 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826581 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826584 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826587 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826590 2565 flags.go:64] FLAG: --lock-file="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826592 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826595 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826598 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826604 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826606 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826609 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826612 2565 flags.go:64] FLAG: --logging-format="text" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826614 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826618 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826620 2565 flags.go:64] FLAG: --manifest-url="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826623 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826627 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826630 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826634 2565 flags.go:64] FLAG: --max-pods="110" Jul 11 03:42:40.833389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826637 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826641 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826644 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826647 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826649 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826653 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826655 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826663 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826666 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826669 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826672 2565 flags.go:64] FLAG: --pod-cidr="" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826675 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826680 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826683 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826687 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826690 2565 flags.go:64] FLAG: --port="10250" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826693 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826696 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ba2256e82009f3d3" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826700 2565 flags.go:64] FLAG: --qos-reserved="" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826703 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826706 2565 flags.go:64] FLAG: --register-node="true" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826709 2565 flags.go:64] FLAG: --register-with-taints="" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826713 2565 flags.go:64] FLAG: --registry-burst="10" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826715 2565 flags.go:64] FLAG: --registry-qps="5" Jul 11 03:42:40.833987 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826718 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826721 2565 flags.go:64] FLAG: --reserved-memory="" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826724 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826727 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826730 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826733 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826736 2565 flags.go:64] FLAG: --runonce="false" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826739 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826742 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826745 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826749 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826752 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826755 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826758 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826761 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826764 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826767 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826770 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826774 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826777 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826780 2565 flags.go:64] FLAG: --system-cgroups="" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826783 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826788 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826791 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826794 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 11 03:42:40.834636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826799 2565 flags.go:64] FLAG: --tls-min-version="" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826801 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826804 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826807 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826809 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826812 2565 flags.go:64] FLAG: --v="2" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826816 2565 flags.go:64] FLAG: --version="false" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826820 2565 flags.go:64] FLAG: --vmodule="" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826824 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.826827 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826951 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826955 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826957 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826960 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826963 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826965 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826968 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826972 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826975 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826977 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826980 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826982 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.835306 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826985 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826987 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826990 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826994 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826997 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.826999 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827002 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827004 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827006 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827010 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827013 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827015 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827018 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827020 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827023 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827027 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827030 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827046 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827049 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827052 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.836132 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827055 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827058 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827061 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827064 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827066 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827069 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827072 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827077 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827080 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827082 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827085 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827088 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827091 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827094 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827097 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827101 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827103 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827106 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827109 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827111 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.836805 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827114 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827117 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827120 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827122 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827125 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827127 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827130 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827132 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827135 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827137 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827140 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827142 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827144 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827147 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827149 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827152 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827155 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827159 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827162 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.837706 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827167 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827169 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827172 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827176 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827179 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827182 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827185 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827189 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827193 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827196 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827198 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827201 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827204 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827207 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827209 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827212 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827215 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827217 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827220 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827222 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.838513 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827225 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827227 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827230 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827232 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827234 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827237 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827239 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.827242 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.828794 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.838481 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.838504 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.838531 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.838546 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838668 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838677 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.839381 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838682 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838688 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838692 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838696 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838700 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838705 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838709 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838714 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838718 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838723 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838727 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838731 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838735 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838739 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838744 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838748 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838753 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838757 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838761 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838765 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.840023 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838769 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838775 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838779 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838784 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838788 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838795 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838801 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838807 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838811 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838816 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838820 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838824 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838829 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838833 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838837 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838841 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838845 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838850 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838854 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.840594 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838858 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838862 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838867 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838871 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838875 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838879 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838884 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838888 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838892 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838897 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838901 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838908 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838914 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838919 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838924 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838928 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838933 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838938 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838942 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838947 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.841254 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838952 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838957 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838962 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838966 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838970 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838975 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838979 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.838983 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839007 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839013 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839018 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839023 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839028 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839051 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839056 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839060 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839065 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839069 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839075 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.841803 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839079 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839083 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839087 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839091 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839095 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839099 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839103 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839107 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839112 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839116 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839121 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839125 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839129 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839136 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839143 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839148 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839152 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839157 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.842495 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839162 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.839170 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839360 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839368 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839373 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839377 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839381 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839386 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839390 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839393 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839397 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839402 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839406 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839412 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 03:42:40.842955 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839419 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839425 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839429 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839434 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839438 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839442 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839446 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839450 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839454 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839459 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839463 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839468 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839472 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839476 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839481 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839485 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839489 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839494 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839498 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839502 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 03:42:40.843558 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839506 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839510 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839515 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839519 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839523 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839527 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839531 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839535 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839539 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839543 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839548 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839552 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839556 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839562 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839566 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839571 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839575 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839579 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839583 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839587 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 03:42:40.844275 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839591 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839595 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839602 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839607 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839611 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839616 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839621 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839625 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839629 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839633 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839638 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839642 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839646 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839651 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839655 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839659 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839663 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839667 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839672 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 03:42:40.844884 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839676 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839680 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839684 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839688 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839692 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839696 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839700 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839706 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839710 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839714 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839718 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839722 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839727 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839731 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839735 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839742 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839748 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839752 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839757 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 03:42:40.845378 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839761 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839765 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839769 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839773 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839778 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839782 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839786 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839790 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:40.839794 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.839803 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.839987 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 11 03:42:40.845857 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.844666 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 11 03:42:40.846433 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.846419 2565 server.go:1019] "Starting client certificate rotation" Jul 11 03:42:40.846547 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.846527 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 11 03:42:40.846585 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.846569 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 11 03:42:40.849300 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.849282 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 11 03:42:40.851569 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.851552 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 11 03:42:40.858210 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.858192 2565 log.go:25] "Validated CRI v1 runtime API" Jul 11 03:42:40.860874 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.860857 2565 log.go:25] "Validated CRI v1 image API" Jul 11 03:42:40.861824 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.861810 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 11 03:42:40.865658 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.865634 2565 fs.go:135] Filesystem UUIDs: map[139bd973-a32d-402a-a8a8-cd663485290e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f61daaca-1243-49bc-af3e-02d185c669a4:/dev/nvme0n1p4] Jul 11 03:42:40.865732 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.865658 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 11 03:42:40.873007 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.872890 2565 manager.go:216] Machine: {Timestamp:2026-07-11 03:42:40.872368158 +0000 UTC m=+0.381624623 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115857 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec293032a85c0c771865db7e4e2c2506 SystemUUID:ec293032-a85c-0c77-1865-db7e4e2c2506 BootID:86912a91-0599-40bc-bdd6-0753c02f4056 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:21:de:93:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:21:de:93:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:a0:d2:13:1e:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 11 03:42:40.873007 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.872991 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 11 03:42:40.873168 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.873089 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 11 03:42:40.874016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.873997 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 11 03:42:40.876191 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.876157 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 11 03:42:40.876334 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.876196 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-240.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 11 03:42:40.877115 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.877105 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 11 03:42:40.877152 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.877118 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 11 03:42:40.877152 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.877127 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 11 03:42:40.877993 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.877982 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 11 03:42:40.879922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.879910 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 11 03:42:40.884578 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884564 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 11 03:42:40.884720 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884710 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 11 03:42:40.884754 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884744 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 11 03:42:40.884754 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884754 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 11 03:42:40.884808 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884771 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 11 03:42:40.884808 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884780 2565 kubelet.go:398] "Adding apiserver pod source" Jul 11 03:42:40.884808 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.884790 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 11 03:42:40.885166 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.885155 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 11 03:42:40.885226 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.885173 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 11 03:42:40.887525 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.887507 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 11 03:42:40.889919 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.889905 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 11 03:42:40.889975 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.889947 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 11 03:42:40.890188 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890178 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 11 03:42:40.890225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890194 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 11 03:42:40.890225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890200 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 11 03:42:40.890225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890205 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 11 03:42:40.890225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890213 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 11 03:42:40.890225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890218 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 11 03:42:40.890795 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890786 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 11 03:42:40.890824 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890798 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 11 03:42:40.890824 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890808 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 11 03:42:40.890824 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890815 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 11 03:42:40.890900 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890828 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 11 03:42:40.890900 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.890839 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 11 03:42:40.892603 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.892587 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 11 03:42:40.892603 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.892603 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 11 03:42:40.892709 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.892580 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-567l8" Jul 11 03:42:40.894389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.894365 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-240.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 11 03:42:40.894502 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.894482 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 11 03:42:40.894641 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.894605 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 11 03:42:40.896420 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.896407 2565 server.go:1268] "Started kubelet" Jul 11 03:42:40.896562 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.896508 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 11 03:42:40.896615 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.896530 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 11 03:42:40.896676 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.896615 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 11 03:42:40.897543 ip-10-0-138-240 systemd[1]: Started Kubernetes Kubelet. Jul 11 03:42:40.899269 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.899247 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 11 03:42:40.899369 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.899247 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 11 03:42:40.899596 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.899561 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-567l8" Jul 11 03:42:40.903005 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.901826 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-240.ec2.internal.18c11f512b663275 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-240.ec2.internal,UID:ip-10-0-138-240.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-240.ec2.internal,},FirstTimestamp:2026-07-11 03:42:40.896356981 +0000 UTC m=+0.405613511,LastTimestamp:2026-07-11 03:42:40.896356981 +0000 UTC m=+0.405613511,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-240.ec2.internal,}" Jul 11 03:42:40.903290 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.903264 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 11 03:42:40.906210 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.906191 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 11 03:42:40.906284 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.906191 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 11 03:42:40.906358 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.906338 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:40.907055 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907016 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 11 03:42:40.907524 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907054 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 11 03:42:40.907524 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907071 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 11 03:42:40.907524 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907182 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 11 03:42:40.907524 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907192 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 11 03:42:40.907791 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907776 2565 factory.go:55] Registering systemd factory Jul 11 03:42:40.907846 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.907794 2565 factory.go:223] Registration of the systemd container factory successfully Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.908367 2565 factory.go:153] Registering CRI-O factory Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.908560 2565 factory.go:223] Registration of the crio container factory successfully Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.908672 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.908700 2565 factory.go:103] Registering Raw factory Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.908720 2565 manager.go:1196] Started watching for new ooms in manager Jul 11 03:42:40.909485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.909208 2565 manager.go:349] Starting recovery of all containers Jul 11 03:42:40.917366 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.917329 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:40.919510 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.919482 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-240.ec2.internal\" not found" node="ip-10-0-138-240.ec2.internal" Jul 11 03:42:40.920193 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.920171 2565 manager.go:354] Recovery completed Jul 11 03:42:40.924614 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.924601 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:40.926777 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.926760 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:40.926832 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.926794 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:40.926832 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.926805 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:40.927383 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.927371 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:40.927439 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.927371 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 11 03:42:40.927439 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.927425 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 11 03:42:40.927491 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.927455 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 11 03:42:40.928142 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.928125 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:40.928194 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.928160 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:40.928194 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.928171 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:40.929715 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.929697 2565 policy_none.go:49] "None policy: Start" Jul 11 03:42:40.929757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.929724 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 11 03:42:40.929757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.929742 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 11 03:42:40.931813 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.931800 2565 policy_none.go:47] "Start" Jul 11 03:42:40.970834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.970808 2565 manager.go:337] "Starting Device Plugin manager" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.970850 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.970864 2565 server.go:87] "Starting device plugin registration server" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.971202 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.971213 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.971322 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.971437 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.971447 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.972001 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.972073 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.982895 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.984193 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.984232 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.984258 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:40.984295 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 11 03:42:40.991172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:40.986121 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:41.071679 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.071591 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:41.072812 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.072797 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:41.072870 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.072830 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:41.072870 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.072844 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:41.072870 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.072866 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.080382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.080367 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.080438 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.080392 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-240.ec2.internal\": node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.084365 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.084327 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal"] Jul 11 03:42:41.084491 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.084392 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:41.085890 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.085873 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:41.085968 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.085906 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:41.085968 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.085918 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:41.088409 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.088396 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:41.088537 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.088523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.088597 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.088560 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:41.089154 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089139 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:41.089233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089162 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:41.089233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089172 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:41.089233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089177 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:41.089233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089203 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:41.089233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.089217 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:41.090957 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.090942 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.091569 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.091551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.091654 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.091583 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 03:42:41.092267 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.092251 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientMemory" Jul 11 03:42:41.092343 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.092276 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 03:42:41.092343 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.092292 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeHasSufficientPID" Jul 11 03:42:41.108328 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.108294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.108328 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.108331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.108517 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.108356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f6d6cfb0ec092ad4ea658444bdb52a5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-240.ec2.internal\" (UID: \"4f6d6cfb0ec092ad4ea658444bdb52a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.109666 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.109650 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-240.ec2.internal\" not found" node="ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.113284 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.113265 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-240.ec2.internal\" not found" node="ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.191255 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.191215 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.209224 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.209224 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.209414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f6d6cfb0ec092ad4ea658444bdb52a5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-240.ec2.internal\" (UID: \"4f6d6cfb0ec092ad4ea658444bdb52a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.209414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f6d6cfb0ec092ad4ea658444bdb52a5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-240.ec2.internal\" (UID: \"4f6d6cfb0ec092ad4ea658444bdb52a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.209414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.209414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.209310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c5f9443c846a247f683d88edebb9b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal\" (UID: \"f6c5f9443c846a247f683d88edebb9b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.292224 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.292184 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.393030 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.392945 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.412451 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.412422 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.418237 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.417998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.493432 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.493387 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.593992 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.593965 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.694569 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.694480 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-240.ec2.internal\" not found" Jul 11 03:42:41.777817 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.777785 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:41.806686 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.806658 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.816543 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.816518 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 11 03:42:41.818438 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.818420 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" Jul 11 03:42:41.826427 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.826405 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 11 03:42:41.845586 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.845548 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 11 03:42:41.845802 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.845776 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 11 03:42:41.845862 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.845778 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 11 03:42:41.845862 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.845778 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 11 03:42:41.885891 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.885862 2565 apiserver.go:52] "Watching apiserver" Jul 11 03:42:41.899757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.899732 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:41.902806 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.902783 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 11 03:42:41.903152 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.903132 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6bn8z","openshift-multus/network-metrics-daemon-2qmkx","openshift-network-diagnostics/network-check-target-rqclv","openshift-network-operator/iptables-alerter-qftbx","openshift-ovn-kubernetes/ovnkube-node-sz6zf","kube-system/global-pull-secret-syncer-bz6lb","kube-system/konnectivity-agent-wcvhn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m","openshift-cluster-node-tuning-operator/tuned-9k6ds","openshift-image-registry/node-ca-p6v5k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal","openshift-multus/multus-additional-cni-plugins-lmsx2","openshift-multus/multus-ghkcs","kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal"] Jul 11 03:42:41.905432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.905414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:41.905693 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.905650 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-10 03:37:42 +0000 UTC" deadline="2028-03-31 01:00:12.13573786 +0000 UTC" Jul 11 03:42:41.905738 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.905697 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15093h17m30.230045166s" Jul 11 03:42:41.906773 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.906759 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 11 03:42:41.907420 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.907401 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 11 03:42:41.907542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.907495 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 11 03:42:41.907609 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.907595 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qxlvb\"" Jul 11 03:42:41.907835 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.907819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.909643 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.909742 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909683 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 11 03:42:41.909847 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.909926 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909911 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8sk7d\"" Jul 11 03:42:41.909986 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 11 03:42:41.910055 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.909911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:41.910113 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.910081 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:41.912589 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-cnibin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912683 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-conf-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912683 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-multus-daemon-config\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912683 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-multus-certs\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912683 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912622 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:41.912683 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-k8s-cni-cncf-io\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-system-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-os-release\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-cni-binary-copy\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-socket-dir-parent\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-multus\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.912904 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-agent-certs\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-konnectivity-ca\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-netns\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-hostroot\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.912984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-etc-kubernetes\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.913008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2nh\" (UniqueName: \"kubernetes.io/projected/eac0671c-6906-4911-9e68-7067cfba69fe-kube-api-access-cj2nh\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.913048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-bin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.913093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.913144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.913125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-kubelet\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:41.915114 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.915084 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 11 03:42:41.915201 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.915141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t58cd\"" Jul 11 03:42:41.917205 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.915380 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:41.917205 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.915667 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.917205 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.916207 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.917205 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.917192 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 11 03:42:41.917946 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.917927 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 11 03:42:41.918053 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.917945 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.918216 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918242 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918265 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918277 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-trblz\"" Jul 11 03:42:41.918377 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918344 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 11 03:42:41.918701 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.918454 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 11 03:42:41.920465 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.920447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:41.922490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.922471 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.922575 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.922473 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.922622 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.922612 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nck22\"" Jul 11 03:42:41.923120 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.923099 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:41.924858 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.924841 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.925013 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.925002 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.925103 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.925090 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sz6cw\"" Jul 11 03:42:41.925409 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.925397 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:41.925464 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:41.925448 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:41.928228 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.928212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:41.930510 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.930493 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.930589 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.930542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:41.930589 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.930577 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g6h7j\"" Jul 11 03:42:41.930798 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.930779 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.930888 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.930807 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 11 03:42:41.932326 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.932307 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 11 03:42:41.932412 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.932367 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 11 03:42:41.932531 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.932518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 11 03:42:41.932779 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.932766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vlc9j\"" Jul 11 03:42:41.933004 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.932987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:41.935239 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.935220 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sdmtl\"" Jul 11 03:42:41.935332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.935224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 11 03:42:41.935332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.935293 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 11 03:42:41.940693 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.940676 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltplf" Jul 11 03:42:41.952144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.952123 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltplf" Jul 11 03:42:41.975208 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:41.975175 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c5f9443c846a247f683d88edebb9b9.slice/crio-45a5b9ca610be6be1232cbda583878da156c1cede05fd0602bb10b7f76c06bff WatchSource:0}: Error finding container 45a5b9ca610be6be1232cbda583878da156c1cede05fd0602bb10b7f76c06bff: Status 404 returned error can't find the container with id 45a5b9ca610be6be1232cbda583878da156c1cede05fd0602bb10b7f76c06bff Jul 11 03:42:41.975429 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:41.975411 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6d6cfb0ec092ad4ea658444bdb52a5.slice/crio-bd3b7db91e4b0e0a019a3d54c8e905ff705bc2280d369ac6742953d4b5649977 WatchSource:0}: Error finding container bd3b7db91e4b0e0a019a3d54c8e905ff705bc2280d369ac6742953d4b5649977: Status 404 returned error can't find the container with id bd3b7db91e4b0e0a019a3d54c8e905ff705bc2280d369ac6742953d4b5649977 Jul 11 03:42:41.979752 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.979735 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 11 03:42:41.989188 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.989129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" event={"ID":"f6c5f9443c846a247f683d88edebb9b9","Type":"ContainerStarted","Data":"45a5b9ca610be6be1232cbda583878da156c1cede05fd0602bb10b7f76c06bff"} Jul 11 03:42:41.991163 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:41.991057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" event={"ID":"4f6d6cfb0ec092ad4ea658444bdb52a5","Type":"ContainerStarted","Data":"bd3b7db91e4b0e0a019a3d54c8e905ff705bc2280d369ac6742953d4b5649977"} Jul 11 03:42:42.007871 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.007849 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 11 03:42:42.013652 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-cnibin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-conf-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-device-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-cnibin\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-tmp\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-cnibin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.013757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-socket-dir-parent\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-conf-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549nn\" (UniqueName: \"kubernetes.io/projected/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-kube-api-access-549nn\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-socket-dir-parent\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9hf\" (UniqueName: \"kubernetes.io/projected/c88bf68b-a2f3-402f-930b-a447a6629b44-kube-api-access-9d9hf\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-etc-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-env-overrides\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-hostroot\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013941 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eb7690c-2a06-447c-8054-917b81c5ae3d-serviceca\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88bf68b-a2f3-402f-930b-a447a6629b44-host-slash\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-config\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.013999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-hostroot\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysconfig\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-var-lib-kubelet\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-kubelet\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-lib-modules\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-multus-daemon-config\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014173 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-multus-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014174 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-kubelet\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n5p\" (UniqueName: \"kubernetes.io/projected/e9dec5b8-a00b-4113-9823-1bf997999428-kube-api-access-v2n5p\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-system-cni-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-systemd-units\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-var-lib-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-ovn\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-system-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-cni-binary-copy\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014408 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-multus\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-tmp-dir\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:42.014594 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-os-release\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-systemd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-multus\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-system-cni-dir\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014543 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-script-lib\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-conf\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-agent-certs\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-konnectivity-ca\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-multus-daemon-config\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-netns\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-etc-kubernetes\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-netns\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-run\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-etc-kubernetes\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89cn\" (UniqueName: \"kubernetes.io/projected/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-kube-api-access-t89cn\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-bin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.015339 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eac0671c-6906-4911-9e68-7067cfba69fe-cni-binary-copy\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-hosts-file\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014931 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-var-lib-cni-bin\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-systemd\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-netns\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.014996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015002 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-modprobe-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-host\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-tuned\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-k8s-cni-cncf-io\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-os-release\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vv6\" (UniqueName: \"kubernetes.io/projected/5eb7690c-2a06-447c-8054-917b81c5ae3d-kube-api-access-54vv6\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88bf68b-a2f3-402f-930b-a447a6629b44-iptables-alerter-script\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-k8s-cni-cncf-io\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-os-release\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016016 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-konnectivity-ca\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-netd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-kubernetes\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2nh\" (UniqueName: \"kubernetes.io/projected/eac0671c-6906-4911-9e68-7067cfba69fe-kube-api-access-cj2nh\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eb7690c-2a06-447c-8054-917b81c5ae3d-host\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-socket-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-sys-fs\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78px\" (UniqueName: \"kubernetes.io/projected/f574859f-4f45-4ca6-9963-eb389f2203b6-kube-api-access-l78px\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp87m\" (UniqueName: \"kubernetes.io/projected/d2c499f0-0864-4fea-9232-ad773e09a1d7-kube-api-access-tp87m\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovn-node-metrics-cert\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-multus-certs\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.016644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-bin\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015934 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-kubelet-config\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-dbus\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.015981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eac0671c-6906-4911-9e68-7067cfba69fe-host-run-multus-certs\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-kubelet\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-node-log\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5n9\" (UniqueName: \"kubernetes.io/projected/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-kube-api-access-gd5n9\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-sys\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-registration-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-slash\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-log-socket\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:42.017107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.016269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:42.018136 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.018120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb06a6fb-10f1-4f74-b5d2-b7c3022068aa-agent-certs\") pod \"konnectivity-agent-wcvhn\" (UID: \"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa\") " pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:42.023311 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:42.023285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2nh\" (UniqueName: \"kubernetes.io/projected/eac0671c-6906-4911-9e68-7067cfba69fe-kube-api-access-cj2nh\") pod \"multus-ghkcs\" (UID: \"eac0671c-6906-4911-9e68-7067cfba69fe\") " pod="openshift-multus/multus-ghkcs" Jul 11 03:42:43.552277 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-registration-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.552277 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-slash\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-log-socket\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-device-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-cnibin\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-slash\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-registration-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-tmp\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-log-socket\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-cnibin\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549nn\" (UniqueName: \"kubernetes.io/projected/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-kube-api-access-549nn\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-device-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9hf\" (UniqueName: \"kubernetes.io/projected/c88bf68b-a2f3-402f-930b-a447a6629b44-kube-api-access-9d9hf\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-etc-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.552518 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-env-overrides\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eb7690c-2a06-447c-8054-917b81c5ae3d-serviceca\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-etc-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88bf68b-a2f3-402f-930b-a447a6629b44-host-slash\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-config\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88bf68b-a2f3-402f-930b-a447a6629b44-host-slash\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysconfig\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-var-lib-kubelet\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-lib-modules\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n5p\" (UniqueName: \"kubernetes.io/projected/e9dec5b8-a00b-4113-9823-1bf997999428-kube-api-access-v2n5p\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-var-lib-kubelet\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-system-cni-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-lib-modules\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.552805 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysconfig\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-system-cni-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-systemd-units\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.553314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-var-lib-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-systemd-units\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553291 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eb7690c-2a06-447c-8054-917b81c5ae3d-serviceca\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-ovn\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-var-lib-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-tmp-dir\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553371 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-ovn\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-os-release\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-systemd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-script-lib\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-conf\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-systemd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553483 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-os-release\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-run\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-env-overrides\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.553563 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89cn\" (UniqueName: \"kubernetes.io/projected/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-kube-api-access-t89cn\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.554196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-hosts-file\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-run\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-sysctl-conf\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.553647 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:42:44.053610471 +0000 UTC m=+2.127226729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-hosts-file\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-config\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-systemd\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-netns\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-systemd\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-modprobe-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-host\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-tuned\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vv6\" (UniqueName: \"kubernetes.io/projected/5eb7690c-2a06-447c-8054-917b81c5ae3d-kube-api-access-54vv6\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-host\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-run-netns\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88bf68b-a2f3-402f-930b-a447a6629b44-iptables-alerter-script\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-netd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-kubernetes\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eb7690c-2a06-447c-8054-917b81c5ae3d-host\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-socket-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.553989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-tmp-dir\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-sys-fs\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-modprobe-d\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554202 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovnkube-script-lib\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78px\" (UniqueName: \"kubernetes.io/projected/f574859f-4f45-4ca6-9963-eb389f2203b6-kube-api-access-l78px\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-sys-fs\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-netd\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554409 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-kubernetes\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:43.555823 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eb7690c-2a06-447c-8054-917b81c5ae3d-host\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp87m\" (UniqueName: \"kubernetes.io/projected/d2c499f0-0864-4fea-9232-ad773e09a1d7-kube-api-access-tp87m\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554524 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovn-node-metrics-cert\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554567 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-socket-dir\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-bin\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c499f0-0864-4fea-9232-ad773e09a1d7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.554841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-cni-bin\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-run-openvswitch\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555097 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88bf68b-a2f3-402f-930b-a447a6629b44-iptables-alerter-script\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.555120 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:43.556314 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.555212 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:44.055195046 +0000 UTC m=+2.128811286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-kubelet-config\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-tmp\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-dbus\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-kubelet\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-kubelet-config\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555377 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-node-log\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-host-kubelet\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5n9\" (UniqueName: \"kubernetes.io/projected/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-kube-api-access-gd5n9\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-node-log\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-dbus\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-sys\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-sys\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c499f0-0864-4fea-9232-ad773e09a1d7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.555764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f574859f-4f45-4ca6-9963-eb389f2203b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.556850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.556763 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-etc-tuned\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.557422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.557152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-ovn-node-metrics-cert\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.560106 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.560078 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:43.560217 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.560110 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:43.560217 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.560125 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:43.560217 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:43.560181 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:44.060164168 +0000 UTC m=+2.133780429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:43.560217 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.560211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9hf\" (UniqueName: \"kubernetes.io/projected/c88bf68b-a2f3-402f-930b-a447a6629b44-kube-api-access-9d9hf\") pod \"iptables-alerter-qftbx\" (UID: \"c88bf68b-a2f3-402f-930b-a447a6629b44\") " pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.560948 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.560923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-549nn\" (UniqueName: \"kubernetes.io/projected/c67a41b4-1865-4ca4-a40c-4bfe16e517b6-kube-api-access-549nn\") pod \"node-resolver-6bn8z\" (UID: \"c67a41b4-1865-4ca4-a40c-4bfe16e517b6\") " pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.561370 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.561350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n5p\" (UniqueName: \"kubernetes.io/projected/e9dec5b8-a00b-4113-9823-1bf997999428-kube-api-access-v2n5p\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:43.562008 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.561987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89cn\" (UniqueName: \"kubernetes.io/projected/b6941d7e-ed28-4d40-88c2-39b0ff8a6d96-kube-api-access-t89cn\") pod \"tuned-9k6ds\" (UID: \"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96\") " pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.562319 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.562304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78px\" (UniqueName: \"kubernetes.io/projected/f574859f-4f45-4ca6-9963-eb389f2203b6-kube-api-access-l78px\") pod \"aws-ebs-csi-driver-node-24t9m\" (UID: \"f574859f-4f45-4ca6-9963-eb389f2203b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.562486 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.562470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp87m\" (UniqueName: \"kubernetes.io/projected/d2c499f0-0864-4fea-9232-ad773e09a1d7-kube-api-access-tp87m\") pod \"multus-additional-cni-plugins-lmsx2\" (UID: \"d2c499f0-0864-4fea-9232-ad773e09a1d7\") " pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.563005 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.562984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vv6\" (UniqueName: \"kubernetes.io/projected/5eb7690c-2a06-447c-8054-917b81c5ae3d-kube-api-access-54vv6\") pod \"node-ca-p6v5k\" (UID: \"5eb7690c-2a06-447c-8054-917b81c5ae3d\") " pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.563005 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.562995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5n9\" (UniqueName: \"kubernetes.io/projected/6177b9e7-f377-426b-9080-5eb8fa3bd5ef-kube-api-access-gd5n9\") pod \"ovnkube-node-sz6zf\" (UID: \"6177b9e7-f377-426b-9080-5eb8fa3bd5ef\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.668774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.668662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:42:43.679512 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.679479 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb06a6fb_10f1_4f74_b5d2_b7c3022068aa.slice/crio-c2526eaa719f8d4689d672072b90041e65e0724efca3b92837e5d4b38cad4176 WatchSource:0}: Error finding container c2526eaa719f8d4689d672072b90041e65e0724efca3b92837e5d4b38cad4176: Status 404 returned error can't find the container with id c2526eaa719f8d4689d672072b90041e65e0724efca3b92837e5d4b38cad4176 Jul 11 03:42:43.680442 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.680411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ghkcs" Jul 11 03:42:43.688779 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.688752 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac0671c_6906_4911_9e68_7067cfba69fe.slice/crio-b2767b6f9e5da2868cfad8570c504daa3d8a4c510a6077b572262799c1a2ae37 WatchSource:0}: Error finding container b2767b6f9e5da2868cfad8570c504daa3d8a4c510a6077b572262799c1a2ae37: Status 404 returned error can't find the container with id b2767b6f9e5da2868cfad8570c504daa3d8a4c510a6077b572262799c1a2ae37 Jul 11 03:42:43.696356 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.696336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qftbx" Jul 11 03:42:43.704740 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.704716 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88bf68b_a2f3_402f_930b_a447a6629b44.slice/crio-f7b1155356f0ce4b4b42b5ef4d65068b3c0028e5c0fa1749aedf5526faa9bef2 WatchSource:0}: Error finding container f7b1155356f0ce4b4b42b5ef4d65068b3c0028e5c0fa1749aedf5526faa9bef2: Status 404 returned error can't find the container with id f7b1155356f0ce4b4b42b5ef4d65068b3c0028e5c0fa1749aedf5526faa9bef2 Jul 11 03:42:43.716638 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.716610 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:42:43.723645 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.723616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" Jul 11 03:42:43.725100 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.725075 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6177b9e7_f377_426b_9080_5eb8fa3bd5ef.slice/crio-ecf658c14c717def77842034e4ea4b02662e1a585d8c064284d12c1965040597 WatchSource:0}: Error finding container ecf658c14c717def77842034e4ea4b02662e1a585d8c064284d12c1965040597: Status 404 returned error can't find the container with id ecf658c14c717def77842034e4ea4b02662e1a585d8c064284d12c1965040597 Jul 11 03:42:43.729462 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.729429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bn8z" Jul 11 03:42:43.732230 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.732191 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6941d7e_ed28_4d40_88c2_39b0ff8a6d96.slice/crio-bc8e21ac73b194e89ffece8fdb9910c80bd45619ea906082c15be111bd31c882 WatchSource:0}: Error finding container bc8e21ac73b194e89ffece8fdb9910c80bd45619ea906082c15be111bd31c882: Status 404 returned error can't find the container with id bc8e21ac73b194e89ffece8fdb9910c80bd45619ea906082c15be111bd31c882 Jul 11 03:42:43.734664 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.734642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6v5k" Jul 11 03:42:43.740164 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.740143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" Jul 11 03:42:43.740455 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.740432 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67a41b4_1865_4ca4_a40c_4bfe16e517b6.slice/crio-b27e23af3a2657fb1eea56909919aa90029a14b523a36d6ae8050221f7686058 WatchSource:0}: Error finding container b27e23af3a2657fb1eea56909919aa90029a14b523a36d6ae8050221f7686058: Status 404 returned error can't find the container with id b27e23af3a2657fb1eea56909919aa90029a14b523a36d6ae8050221f7686058 Jul 11 03:42:43.744825 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:43.744809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" Jul 11 03:42:43.745211 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.745184 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb7690c_2a06_447c_8054_917b81c5ae3d.slice/crio-9bb06815fd23687e44593f16e58699fd58f69571d0a56e4681e9ea127297cc4a WatchSource:0}: Error finding container 9bb06815fd23687e44593f16e58699fd58f69571d0a56e4681e9ea127297cc4a: Status 404 returned error can't find the container with id 9bb06815fd23687e44593f16e58699fd58f69571d0a56e4681e9ea127297cc4a Jul 11 03:42:43.750569 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.750301 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf574859f_4f45_4ca6_9963_eb389f2203b6.slice/crio-167bcce79586412bd0ca2a5f1528663fc81e09408ceed31298e4f3d293e8f9f7 WatchSource:0}: Error finding container 167bcce79586412bd0ca2a5f1528663fc81e09408ceed31298e4f3d293e8f9f7: Status 404 returned error can't find the container with id 167bcce79586412bd0ca2a5f1528663fc81e09408ceed31298e4f3d293e8f9f7 Jul 11 03:42:43.755111 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:42:43.755087 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c499f0_0864_4fea_9232_ad773e09a1d7.slice/crio-fa1ede2433d7f436e5e7fa4ce079a0cc961d6ca015ac53c9c7e702ba737641e3 WatchSource:0}: Error finding container fa1ede2433d7f436e5e7fa4ce079a0cc961d6ca015ac53c9c7e702ba737641e3: Status 404 returned error can't find the container with id fa1ede2433d7f436e5e7fa4ce079a0cc961d6ca015ac53c9c7e702ba737641e3 Jul 11 03:42:44.063394 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.063309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:44.063394 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.063385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.063423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063476 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063549 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.06352872 +0000 UTC m=+3.137144959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063554 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063574 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:44.063603 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063587 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:44.063884 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063641 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.063624149 +0000 UTC m=+3.137240411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:44.063884 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063731 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:44.063884 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:44.063766 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:42:45.063754137 +0000 UTC m=+3.137370391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:44.391259 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.391190 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 03:37:43 +0000 UTC" deadline="2027-12-16 20:42:32.837776879 +0000 UTC" Jul 11 03:42:44.391259 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.391257 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12568h59m48.446524351s" Jul 11 03:42:44.434653 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.434613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerStarted","Data":"fa1ede2433d7f436e5e7fa4ce079a0cc961d6ca015ac53c9c7e702ba737641e3"} Jul 11 03:42:44.461746 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.461704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" event={"ID":"f574859f-4f45-4ca6-9963-eb389f2203b6","Type":"ContainerStarted","Data":"167bcce79586412bd0ca2a5f1528663fc81e09408ceed31298e4f3d293e8f9f7"} Jul 11 03:42:44.470042 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.470002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghkcs" event={"ID":"eac0671c-6906-4911-9e68-7067cfba69fe","Type":"ContainerStarted","Data":"b2767b6f9e5da2868cfad8570c504daa3d8a4c510a6077b572262799c1a2ae37"} Jul 11 03:42:44.477826 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.477754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6v5k" event={"ID":"5eb7690c-2a06-447c-8054-917b81c5ae3d","Type":"ContainerStarted","Data":"9bb06815fd23687e44593f16e58699fd58f69571d0a56e4681e9ea127297cc4a"} Jul 11 03:42:44.501399 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.501359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bn8z" event={"ID":"c67a41b4-1865-4ca4-a40c-4bfe16e517b6","Type":"ContainerStarted","Data":"b27e23af3a2657fb1eea56909919aa90029a14b523a36d6ae8050221f7686058"} Jul 11 03:42:44.508588 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.508473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" event={"ID":"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96","Type":"ContainerStarted","Data":"bc8e21ac73b194e89ffece8fdb9910c80bd45619ea906082c15be111bd31c882"} Jul 11 03:42:44.526737 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.526699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"ecf658c14c717def77842034e4ea4b02662e1a585d8c064284d12c1965040597"} Jul 11 03:42:44.534549 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.534513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qftbx" event={"ID":"c88bf68b-a2f3-402f-930b-a447a6629b44","Type":"ContainerStarted","Data":"f7b1155356f0ce4b4b42b5ef4d65068b3c0028e5c0fa1749aedf5526faa9bef2"} Jul 11 03:42:44.547580 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.547527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcvhn" event={"ID":"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa","Type":"ContainerStarted","Data":"c2526eaa719f8d4689d672072b90041e65e0724efca3b92837e5d4b38cad4176"} Jul 11 03:42:44.619158 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.619108 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:44.826983 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:44.826888 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:45.072821 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.072116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:45.072821 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.072411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:45.072821 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.072445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:45.072821 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.072628 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:45.072821 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.072707 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:47.072672013 +0000 UTC m=+5.146288264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:45.073218 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073113 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:45.073218 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073161 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:42:47.07314725 +0000 UTC m=+5.146763502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:45.073325 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073249 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:45.073325 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073267 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:45.073325 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073279 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:45.073466 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.073315 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:47.073304134 +0000 UTC m=+5.146920386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:45.256095 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.255859 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 03:42:45.392418 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.392371 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 03:37:43 +0000 UTC" deadline="2028-04-17 06:30:04.241355154 +0000 UTC" Jul 11 03:42:45.392418 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.392413 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15506h47m18.848946006s" Jul 11 03:42:45.420902 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.420869 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:45.421081 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.421000 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:45.421396 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.421372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:45.421504 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.421475 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:45.421574 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:45.421554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:45.421733 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:45.421628 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.093625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.093698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.093762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.093895 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.093961 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.093971 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:42:51.093951309 +0000 UTC m=+9.167567561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.093978 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.093990 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.094000 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.094024 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:51.09401466 +0000 UTC m=+9.167630902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:47.094318 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.094069 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:51.09405023 +0000 UTC m=+9.167666484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:47.420421 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.420386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:47.420588 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.420525 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:47.420945 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.420925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:47.421058 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.421036 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:47.421125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:47.421103 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:47.421186 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:47.421169 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:49.420804 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:49.420767 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:49.420804 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:49.420806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:49.421312 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:49.420775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:49.421312 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:49.420895 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:49.421312 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:49.421011 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:49.421312 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:49.421059 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.128811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.128892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.128930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.128953 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129011 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:42:59.128997407 +0000 UTC m=+17.202613643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129056 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129108 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:59.129092355 +0000 UTC m=+17.202708606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129185 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129202 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129213 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:51.129273 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.129250 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:42:59.129237018 +0000 UTC m=+17.202853255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.420212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.420240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:51.420333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.420342 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.420446 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:51.420642 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:51.420522 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:53.421310 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:53.421276 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:53.421310 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:53.421280 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:53.421850 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:53.421422 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:53.421850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:53.421758 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:53.421951 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:53.421847 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:53.421951 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:53.421932 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:55.420272 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:55.420231 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:55.420766 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:55.420232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:55.420766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:55.420380 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:55.420766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:55.420466 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:55.420766 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:55.420233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:55.420766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:55.420582 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:57.420313 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:57.420334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:57.420334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:57.420452 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:57.420538 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:57.420852 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:57.420619 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:42:59.189352 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.189318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.189368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.189401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189480 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189486 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189552 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:15.189532963 +0000 UTC m=+33.263149200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189559 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189581 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189593 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189568 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:43:15.18955963 +0000 UTC m=+33.263175868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:42:59.189766 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.189643 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:15.189631595 +0000 UTC m=+33.263247832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:42:59.420258 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.420230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:42:59.420437 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.420330 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:42:59.420437 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.420334 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:42:59.420533 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.420437 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:42:59.420533 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:42:59.420479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:42:59.420609 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:42:59.420552 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:01.421248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:01.421208 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:01.421248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:01.421248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:01.421810 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:01.421248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:01.421810 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:01.421349 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:01.421810 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:01.421436 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:01.421810 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:01.421536 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:03.420592 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:03.420551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:03.420592 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:03.420583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:03.421124 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:03.420551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:03.421124 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:03.420696 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:03.421124 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:03.420789 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:03.421124 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:03.420861 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:04.596906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.596597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ghkcs" event={"ID":"eac0671c-6906-4911-9e68-7067cfba69fe","Type":"ContainerStarted","Data":"1859ea97f9bf9c9498f418e504643f6046b30a0426557233139ab997e1b426f9"} Jul 11 03:43:04.598428 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.598401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" event={"ID":"4f6d6cfb0ec092ad4ea658444bdb52a5","Type":"ContainerStarted","Data":"4f77bb5da320e11fe129e72e414ca982bfdf110bcbc56a2a9b767bc85bbcfe52"} Jul 11 03:43:04.599962 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.599940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" event={"ID":"b6941d7e-ed28-4d40-88c2-39b0ff8a6d96","Type":"ContainerStarted","Data":"e803c6e0ed1d38decfafdf4a73c004944a9afec33f4226895709615ce6e53be0"} Jul 11 03:43:04.612492 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.612446 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ghkcs" podStartSLOduration=2.111138781 podStartE2EDuration="22.612430644s" totalImagesPullingTime="20.501291863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:04.611717495 +0000 UTC m=+22.685333755" watchObservedRunningTime="2026-07-11 03:43:04.612430644 +0000 UTC m=+22.686046902" Jul 11 03:43:04.625887 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.625812 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9k6ds" podStartSLOduration=2.388132354 podStartE2EDuration="22.625793442s" totalImagesPullingTime="20.237661088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:04.625577934 +0000 UTC m=+22.699194194" watchObservedRunningTime="2026-07-11 03:43:04.625793442 +0000 UTC m=+22.699409702" Jul 11 03:43:04.639038 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:04.638985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-240.ec2.internal" podStartSLOduration=21.638972976 podStartE2EDuration="21.638972976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:04.638904001 +0000 UTC m=+22.712520259" watchObservedRunningTime="2026-07-11 03:43:04.638972976 +0000 UTC m=+22.712589234" Jul 11 03:43:05.420963 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.420796 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:05.421077 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.420799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:05.421077 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.420849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:05.421142 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:05.421115 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:05.421217 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:05.421202 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:05.421281 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:05.421260 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:05.604617 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.604590 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.604933 2565 generic.go:356] "Generic (PLEG): container finished" podID="6177b9e7-f377-426b-9080-5eb8fa3bd5ef" containerID="768eb9a4ebb63c2ad3cf63387b7456b3d71ddb0f4889c82e6b80643dbedfe3d3" exitCode=1 Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605007 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"43d7f1c05f271434cd5eeda8a97d2be0c9bc789dc491deef01e6bd328b405b2a"} Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"ce82005512e0b3ed92b6897fa8011e6e49d97abd838340f1581a75f07912cba4"} Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"d7b27eb4cc717f4d47c0a0f2bd9333c5d2db3e652a8071455ff3e753d3a522ac"} Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"cb38546818f3376d1b201bf7b9fbfdd371f0ee99b2412b5f570c3afad72910c5"} Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerDied","Data":"768eb9a4ebb63c2ad3cf63387b7456b3d71ddb0f4889c82e6b80643dbedfe3d3"} Jul 11 03:43:05.605287 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.605228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"8efde7f6a1472c73b4e845c19b80de9eb0b0ce852aec6db1fbcd09fc61ccb33d"} Jul 11 03:43:05.606484 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.606413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qftbx" event={"ID":"c88bf68b-a2f3-402f-930b-a447a6629b44","Type":"ContainerStarted","Data":"ea661825c460542833ce4e0745d3e915bd3b02484c98c8bc5fa96c44b332b1c5"} Jul 11 03:43:05.607825 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.607805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcvhn" event={"ID":"cb06a6fb-10f1-4f74-b5d2-b7c3022068aa","Type":"ContainerStarted","Data":"2e855e10ef9c9ffccfaefb28962b13085139053e6786c2e7c7743023bf315354"} Jul 11 03:43:05.609233 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.609213 2565 generic.go:356] "Generic (PLEG): container finished" podID="f6c5f9443c846a247f683d88edebb9b9" containerID="70f89fafe15496ef96c6433eb544286daa12eaf85e827147f96615baad0174e4" exitCode=0 Jul 11 03:43:05.609311 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.609276 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" event={"ID":"f6c5f9443c846a247f683d88edebb9b9","Type":"ContainerDied","Data":"70f89fafe15496ef96c6433eb544286daa12eaf85e827147f96615baad0174e4"} Jul 11 03:43:05.610649 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.610628 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="3f9e3d05286f381e1f446c6e3aef82059af3f8a06126b7bcfe123a7e0f704ffd" exitCode=0 Jul 11 03:43:05.610738 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.610708 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"3f9e3d05286f381e1f446c6e3aef82059af3f8a06126b7bcfe123a7e0f704ffd"} Jul 11 03:43:05.612216 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.612181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" event={"ID":"f574859f-4f45-4ca6-9963-eb389f2203b6","Type":"ContainerStarted","Data":"3b3de811a51f8877135c7c110a411bbd3eb7952039ec58f1ceab7d09e254c5f7"} Jul 11 03:43:05.613533 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.613514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6v5k" event={"ID":"5eb7690c-2a06-447c-8054-917b81c5ae3d","Type":"ContainerStarted","Data":"92ad307ca47e2d7d710914e277b70cb5648a5e6a6d3422ef9bb52cba483e3eca"} Jul 11 03:43:05.614991 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.614967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bn8z" event={"ID":"c67a41b4-1865-4ca4-a40c-4bfe16e517b6","Type":"ContainerStarted","Data":"bb921d31ff4ba641fa4cf1d472053c98c599ae96ec1bff387e200ecb3b6ef6ec"} Jul 11 03:43:05.632427 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.632369 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcvhn" podStartSLOduration=3.34265384 podStartE2EDuration="23.632349188s" totalImagesPullingTime="20.289695348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:05.632057327 +0000 UTC m=+23.705673586" watchObservedRunningTime="2026-07-11 03:43:05.632349188 +0000 UTC m=+23.705965447" Jul 11 03:43:05.633048 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.633016 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qftbx" podStartSLOduration=3.362407026 podStartE2EDuration="23.63299999s" totalImagesPullingTime="20.270592964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:05.619599354 +0000 UTC m=+23.693215613" watchObservedRunningTime="2026-07-11 03:43:05.63299999 +0000 UTC m=+23.706616246" Jul 11 03:43:05.645207 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.645159 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6bn8z" podStartSLOduration=3.426890461 podStartE2EDuration="23.645144534s" totalImagesPullingTime="20.218254073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:05.644840797 +0000 UTC m=+23.718457056" watchObservedRunningTime="2026-07-11 03:43:05.645144534 +0000 UTC m=+23.718760789" Jul 11 03:43:05.675921 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:05.675864 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6v5k" podStartSLOduration=3.462798363 podStartE2EDuration="23.675848462s" totalImagesPullingTime="20.213050099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:05.675597451 +0000 UTC m=+23.749213752" watchObservedRunningTime="2026-07-11 03:43:05.675848462 +0000 UTC m=+23.749464721" Jul 11 03:43:06.405937 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:06.405902 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 11 03:43:06.420901 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:06.420789 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-11T03:43:06.405932858Z","UUID":"18468186-eeea-4734-acdd-b7b025619bbd","Handler":null,"Name":"","Endpoint":""} Jul 11 03:43:06.422569 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:06.422437 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 11 03:43:06.422569 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:06.422484 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 11 03:43:06.619505 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:06.619419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" event={"ID":"f574859f-4f45-4ca6-9963-eb389f2203b6","Type":"ContainerStarted","Data":"20b056b392ad286baf081ac70c515d86dc8816cfbdf0bcecd274cfaa96e2cfb7"} Jul 11 03:43:07.420328 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:07.420283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:07.420328 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:07.420315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:07.420692 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:07.420315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:07.420692 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:07.420432 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:07.420692 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:07.420557 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:07.420692 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:07.420637 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:07.622972 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:07.622943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" event={"ID":"f6c5f9443c846a247f683d88edebb9b9","Type":"ContainerStarted","Data":"84ad6ec1480549bb7a9fcc590d975d420750798303249df0560778c53d431aab"} Jul 11 03:43:07.644169 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:07.644109 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-240.ec2.internal" podStartSLOduration=24.644094758 podStartE2EDuration="24.644094758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:07.643943376 +0000 UTC m=+25.717559635" watchObservedRunningTime="2026-07-11 03:43:07.644094758 +0000 UTC m=+25.717711016" Jul 11 03:43:08.627841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:08.627801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" event={"ID":"f574859f-4f45-4ca6-9963-eb389f2203b6","Type":"ContainerStarted","Data":"9dd6c4ca6872b22a5760125731dba823040769b4b22138980b97b9fdd2a3779c"} Jul 11 03:43:08.631688 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:08.631642 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:43:08.632094 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:08.632063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"69deea4236efeb7b393ed69b2a2c3d8d1d810f3eb56b410aa432488f071e2dc8"} Jul 11 03:43:08.644304 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:08.644261 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24t9m" podStartSLOduration=4.097526092 podStartE2EDuration="26.644244707s" totalImagesPullingTime="22.546718615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:08.643275168 +0000 UTC m=+26.716891429" watchObservedRunningTime="2026-07-11 03:43:08.644244707 +0000 UTC m=+26.717860967" Jul 11 03:43:09.420376 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:09.420342 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:09.420376 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:09.420359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:09.420633 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:09.420364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:09.420633 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:09.420473 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:09.420633 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:09.420558 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:09.420815 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:09.420639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:10.116317 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.116237 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:43:10.117006 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.116989 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:43:10.641402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.641237 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:43:10.641770 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.641742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"354d5b64d271ca4a1dfb221c9ff208c1a05bb333ca6d394911ab5c68a532b019"} Jul 11 03:43:10.642031 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.642004 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:10.642252 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.642089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:10.642252 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.642134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:10.642252 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.642243 2565 scope.go:117] "RemoveContainer" containerID="768eb9a4ebb63c2ad3cf63387b7456b3d71ddb0f4889c82e6b80643dbedfe3d3" Jul 11 03:43:10.644283 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.644258 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="5c4e0ded175b5fb2474e5703b38d6d121bf582937aca6372ec183f2b330e6149" exitCode=0 Jul 11 03:43:10.644399 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.644352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"5c4e0ded175b5fb2474e5703b38d6d121bf582937aca6372ec183f2b330e6149"} Jul 11 03:43:10.644457 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.644441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:43:10.645179 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.645162 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcvhn" Jul 11 03:43:10.658817 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.658796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:10.659280 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:10.659268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:11.420298 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.420266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:11.420784 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.420277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:11.420784 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:11.420377 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:11.420784 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.420391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:11.420784 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:11.420496 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:11.420784 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:11.420574 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:11.649971 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.649945 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:43:11.650316 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.650292 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" event={"ID":"6177b9e7-f377-426b-9080-5eb8fa3bd5ef","Type":"ContainerStarted","Data":"d3aebc77aacc23a8b3ef1fff0d0edc382da00e7a99b50b6374df096f66547033"} Jul 11 03:43:11.675057 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:11.674952 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" podStartSLOduration=8.735361971 podStartE2EDuration="29.674931245s" totalImagesPullingTime="20.939569274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:11.674753265 +0000 UTC m=+29.748369524" watchObservedRunningTime="2026-07-11 03:43:11.674931245 +0000 UTC m=+29.748547689" Jul 11 03:43:12.259542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.259312 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rqclv"] Jul 11 03:43:12.259774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.259569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:12.259774 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:12.259654 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:12.262329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.262299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz6lb"] Jul 11 03:43:12.262473 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.262441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:12.262581 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:12.262557 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:12.263012 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.262986 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2qmkx"] Jul 11 03:43:12.263149 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.263094 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:12.263226 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:12.263202 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:12.655057 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.655021 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="51181a1f42dffafa1db5c0ba2946d4470f13ea0ee392a492068aa9b7b7eb7b2f" exitCode=0 Jul 11 03:43:12.655644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:12.655113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"51181a1f42dffafa1db5c0ba2946d4470f13ea0ee392a492068aa9b7b7eb7b2f"} Jul 11 03:43:13.420870 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:13.420832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:13.421033 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:13.420939 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:14.420656 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:14.420618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:14.420656 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:14.420640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:14.421402 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:14.420747 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:14.421402 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:14.420900 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:14.661897 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:14.661862 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="fc87c0e9e15a6f4bfd9fdb3b6a6026e155858369e96fbe1fd250ed4d0491db97" exitCode=0 Jul 11 03:43:14.662089 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:14.661921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"fc87c0e9e15a6f4bfd9fdb3b6a6026e155858369e96fbe1fd250ed4d0491db97"} Jul 11 03:43:15.208780 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:15.208741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:15.208791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:15.208817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208910 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208917 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208922 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208948 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208962 2565 projected.go:196] Error preparing data for projected volume kube-api-access-6llln for pod openshift-network-diagnostics/network-check-target-rqclv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.208982 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs podName:e9dec5b8-a00b-4113-9823-1bf997999428 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:47.208963107 +0000 UTC m=+65.282579347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs") pod "network-metrics-daemon-2qmkx" (UID: "e9dec5b8-a00b-4113-9823-1bf997999428") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 03:43:15.209008 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.209001 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret podName:2de47295-6cd6-4b77-a057-6d8f5d1b22bd nodeName:}" failed. No retries permitted until 2026-07-11 03:43:47.208991149 +0000 UTC m=+65.282607388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret") pod "global-pull-secret-syncer-bz6lb" (UID: "2de47295-6cd6-4b77-a057-6d8f5d1b22bd") : object "kube-system"/"original-pull-secret" not registered Jul 11 03:43:15.209500 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.209016 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln podName:34cf96ad-9684-4763-a44e-541ab12906e7 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:47.209007929 +0000 UTC m=+65.282624172 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6llln" (UniqueName: "kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln") pod "network-check-target-rqclv" (UID: "34cf96ad-9684-4763-a44e-541ab12906e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 03:43:15.420320 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:15.420289 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:15.420548 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:15.420441 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz6lb" podUID="2de47295-6cd6-4b77-a057-6d8f5d1b22bd" Jul 11 03:43:16.420307 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:16.420273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:16.420780 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:16.420405 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rqclv" podUID="34cf96ad-9684-4763-a44e-541ab12906e7" Jul 11 03:43:16.420780 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:16.420442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:16.420780 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:16.420561 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2qmkx" podUID="e9dec5b8-a00b-4113-9823-1bf997999428" Jul 11 03:43:17.211812 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.211546 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-240.ec2.internal" event="NodeReady" Jul 11 03:43:17.211977 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.211887 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 11 03:43:17.255822 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.255766 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nn9nn"] Jul 11 03:43:17.259484 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.259448 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j5h5d"] Jul 11 03:43:17.259628 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.259604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.262125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.261860 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 11 03:43:17.262125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.261881 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 11 03:43:17.262125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.261991 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9qtgh\"" Jul 11 03:43:17.262125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.262053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.263992 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.263972 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 11 03:43:17.264244 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.264228 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m57wb\"" Jul 11 03:43:17.264432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.264416 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 11 03:43:17.264629 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.264613 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 11 03:43:17.267197 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.267172 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nn9nn"] Jul 11 03:43:17.267759 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.267739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j5h5d"] Jul 11 03:43:17.420571 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.420546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:17.422730 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.422708 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 11 03:43:17.424561 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a39043c-ddee-4dbc-94b8-d3e387668742-config-volume\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.424669 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.424669 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a39043c-ddee-4dbc-94b8-d3e387668742-tmp-dir\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.424669 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntc8\" (UniqueName: \"kubernetes.io/projected/1d17d352-eb89-4c0e-8b78-71d76d5b567d-kube-api-access-cntc8\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.424838 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4jf\" (UniqueName: \"kubernetes.io/projected/4a39043c-ddee-4dbc-94b8-d3e387668742-kube-api-access-pg4jf\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.424838 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.424809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17d352-eb89-4c0e-8b78-71d76d5b567d-cert\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.525783 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.525693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.525783 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.525744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a39043c-ddee-4dbc-94b8-d3e387668742-tmp-dir\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.526023 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:17.525875 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 03:43:17.526023 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.525891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntc8\" (UniqueName: \"kubernetes.io/projected/1d17d352-eb89-4c0e-8b78-71d76d5b567d-kube-api-access-cntc8\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.526023 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:17.525959 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls podName:4a39043c-ddee-4dbc-94b8-d3e387668742 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:18.025933902 +0000 UTC m=+36.099550151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls") pod "dns-default-nn9nn" (UID: "4a39043c-ddee-4dbc-94b8-d3e387668742") : secret "dns-default-metrics-tls" not found Jul 11 03:43:17.526171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.526037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4jf\" (UniqueName: \"kubernetes.io/projected/4a39043c-ddee-4dbc-94b8-d3e387668742-kube-api-access-pg4jf\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.526171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.526065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17d352-eb89-4c0e-8b78-71d76d5b567d-cert\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.526171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.526106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a39043c-ddee-4dbc-94b8-d3e387668742-tmp-dir\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.526171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.526114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a39043c-ddee-4dbc-94b8-d3e387668742-config-volume\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.526914 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.526890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4a39043c-ddee-4dbc-94b8-d3e387668742-config-volume\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.531243 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.531221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d17d352-eb89-4c0e-8b78-71d76d5b567d-cert\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.534029 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.534005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4jf\" (UniqueName: \"kubernetes.io/projected/4a39043c-ddee-4dbc-94b8-d3e387668742-kube-api-access-pg4jf\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:17.534336 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.534311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntc8\" (UniqueName: \"kubernetes.io/projected/1d17d352-eb89-4c0e-8b78-71d76d5b567d-kube-api-access-cntc8\") pod \"ingress-canary-j5h5d\" (UID: \"1d17d352-eb89-4c0e-8b78-71d76d5b567d\") " pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.579365 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.579327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j5h5d" Jul 11 03:43:17.731475 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:17.731438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j5h5d"] Jul 11 03:43:18.031143 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.031100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:18.031313 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:18.031268 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 03:43:18.031354 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:18.031345 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls podName:4a39043c-ddee-4dbc-94b8-d3e387668742 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:19.031327365 +0000 UTC m=+37.104943606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls") pod "dns-default-nn9nn" (UID: "4a39043c-ddee-4dbc-94b8-d3e387668742") : secret "dns-default-metrics-tls" not found Jul 11 03:43:18.420362 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.420329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:18.420362 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.420358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:18.422891 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.422853 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 11 03:43:18.423275 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.423099 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 11 03:43:18.423911 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.423692 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6d8m4\"" Jul 11 03:43:18.423911 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.423719 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-57qws\"" Jul 11 03:43:18.423911 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:18.423721 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 11 03:43:19.036984 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.036952 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jjsbf"] Jul 11 03:43:19.038122 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.038087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:19.038255 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:19.038239 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 03:43:19.038321 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:19.038300 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls podName:4a39043c-ddee-4dbc-94b8-d3e387668742 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:21.038283282 +0000 UTC m=+39.111899536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls") pod "dns-default-nn9nn" (UID: "4a39043c-ddee-4dbc-94b8-d3e387668742") : secret "dns-default-metrics-tls" not found Jul 11 03:43:19.042126 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.042104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.044138 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.044113 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 11 03:43:19.045803 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.045254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 11 03:43:19.045803 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.045311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 11 03:43:19.045803 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.045422 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 11 03:43:19.045803 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.045456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7bl4s\"" Jul 11 03:43:19.049610 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.048531 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jjsbf"] Jul 11 03:43:19.138668 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.138629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-key\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.138842 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.138727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7xb\" (UniqueName: \"kubernetes.io/projected/286f552e-7fa3-4072-8b2b-19861a3a33d4-kube-api-access-jp7xb\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.138842 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.138773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-cabundle\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.168691 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:19.168640 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d17d352_eb89_4c0e_8b78_71d76d5b567d.slice/crio-9a98ec965d05a21138239e460d117b0a04aa367b4af2ed5d9de081f7d139e15f WatchSource:0}: Error finding container 9a98ec965d05a21138239e460d117b0a04aa367b4af2ed5d9de081f7d139e15f: Status 404 returned error can't find the container with id 9a98ec965d05a21138239e460d117b0a04aa367b4af2ed5d9de081f7d139e15f Jul 11 03:43:19.240091 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.240071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-key\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.240183 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.240120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7xb\" (UniqueName: \"kubernetes.io/projected/286f552e-7fa3-4072-8b2b-19861a3a33d4-kube-api-access-jp7xb\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.240183 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.240151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-cabundle\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.240850 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.240827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-cabundle\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.243187 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.243164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/286f552e-7fa3-4072-8b2b-19861a3a33d4-signing-key\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.248718 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.248695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7xb\" (UniqueName: \"kubernetes.io/projected/286f552e-7fa3-4072-8b2b-19861a3a33d4-kube-api-access-jp7xb\") pod \"service-ca-79f68b9f4f-jjsbf\" (UID: \"286f552e-7fa3-4072-8b2b-19861a3a33d4\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.354312 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.354117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" Jul 11 03:43:19.503977 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.503948 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jjsbf"] Jul 11 03:43:19.507314 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:19.507282 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286f552e_7fa3_4072_8b2b_19861a3a33d4.slice/crio-81b6cf5d1065d6d479b2f92f6b7d72e453efe7a012f781efc52868edf59f2003 WatchSource:0}: Error finding container 81b6cf5d1065d6d479b2f92f6b7d72e453efe7a012f781efc52868edf59f2003: Status 404 returned error can't find the container with id 81b6cf5d1065d6d479b2f92f6b7d72e453efe7a012f781efc52868edf59f2003 Jul 11 03:43:19.676337 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.676289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" event={"ID":"286f552e-7fa3-4072-8b2b-19861a3a33d4","Type":"ContainerStarted","Data":"81b6cf5d1065d6d479b2f92f6b7d72e453efe7a012f781efc52868edf59f2003"} Jul 11 03:43:19.679404 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.679367 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="406bf6c85e19d610bc758c534996290b1cfbad34991acf1168d8e9fefd91d348" exitCode=0 Jul 11 03:43:19.679533 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.679429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"406bf6c85e19d610bc758c534996290b1cfbad34991acf1168d8e9fefd91d348"} Jul 11 03:43:19.680871 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:19.680839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j5h5d" event={"ID":"1d17d352-eb89-4c0e-8b78-71d76d5b567d","Type":"ContainerStarted","Data":"9a98ec965d05a21138239e460d117b0a04aa367b4af2ed5d9de081f7d139e15f"} Jul 11 03:43:20.687612 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:20.687574 2565 generic.go:356] "Generic (PLEG): container finished" podID="d2c499f0-0864-4fea-9232-ad773e09a1d7" containerID="120cacbb56806b268ddb6ff643d19bb31ec42cb6208301833468e551054d15ed" exitCode=0 Jul 11 03:43:20.688349 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:20.687635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerDied","Data":"120cacbb56806b268ddb6ff643d19bb31ec42cb6208301833468e551054d15ed"} Jul 11 03:43:21.058846 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:21.058744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:21.058980 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:21.058922 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 03:43:21.059031 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:21.059003 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls podName:4a39043c-ddee-4dbc-94b8-d3e387668742 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:25.058980146 +0000 UTC m=+43.132596395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls") pod "dns-default-nn9nn" (UID: "4a39043c-ddee-4dbc-94b8-d3e387668742") : secret "dns-default-metrics-tls" not found Jul 11 03:43:21.694290 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:21.694258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" event={"ID":"d2c499f0-0864-4fea-9232-ad773e09a1d7","Type":"ContainerStarted","Data":"50d4eb64985b6a4738315d14ef31995080a441a361ea0a83ca20a29116fc4f73"} Jul 11 03:43:21.717390 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:21.716747 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lmsx2" podStartSLOduration=7.922266413 podStartE2EDuration="39.716725608s" totalImagesPullingTime="31.794459195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:21.714953065 +0000 UTC m=+39.788569341" watchObservedRunningTime="2026-07-11 03:43:21.716725608 +0000 UTC m=+39.790341868" Jul 11 03:43:22.699139 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:22.699101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" event={"ID":"286f552e-7fa3-4072-8b2b-19861a3a33d4","Type":"ContainerStarted","Data":"41e77d358ca682aa24e17b797be23b055a56e2798a3c9dea04b2a8a7c523d663"} Jul 11 03:43:22.700746 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:22.700713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j5h5d" event={"ID":"1d17d352-eb89-4c0e-8b78-71d76d5b567d","Type":"ContainerStarted","Data":"cf0236f30a21c99ffdf8c830d80ff00c472b856553511c97910cf584946e3413"} Jul 11 03:43:23.718262 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:23.718203 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-jjsbf" podStartSLOduration=1.659193765 podStartE2EDuration="4.718181676s" totalImagesPullingTime="3.058987911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:23.717663946 +0000 UTC m=+41.791280203" watchObservedRunningTime="2026-07-11 03:43:23.718181676 +0000 UTC m=+41.791797937" Jul 11 03:43:23.718763 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:23.718561 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j5h5d" podStartSLOduration=3.326083657 podStartE2EDuration="6.718552258s" totalImagesPullingTime="3.392468601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:22.716310966 +0000 UTC m=+40.789927224" watchObservedRunningTime="2026-07-11 03:43:23.718552258 +0000 UTC m=+41.792168519" Jul 11 03:43:25.090363 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:25.090311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:25.090752 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:25.090423 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 03:43:25.090752 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:25.090498 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls podName:4a39043c-ddee-4dbc-94b8-d3e387668742 nodeName:}" failed. No retries permitted until 2026-07-11 03:43:33.090480585 +0000 UTC m=+51.164096822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls") pod "dns-default-nn9nn" (UID: "4a39043c-ddee-4dbc-94b8-d3e387668742") : secret "dns-default-metrics-tls" not found Jul 11 03:43:33.152730 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:33.152661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:33.156191 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:33.156156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a39043c-ddee-4dbc-94b8-d3e387668742-metrics-tls\") pod \"dns-default-nn9nn\" (UID: \"4a39043c-ddee-4dbc-94b8-d3e387668742\") " pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:33.173659 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:33.173623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:33.316926 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:33.316892 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nn9nn"] Jul 11 03:43:33.320361 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:33.320333 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a39043c_ddee_4dbc_94b8_d3e387668742.slice/crio-20590919bf7b0abe73aa3b715acf6dd2ada4acbe612d8147b956e78ae0c75939 WatchSource:0}: Error finding container 20590919bf7b0abe73aa3b715acf6dd2ada4acbe612d8147b956e78ae0c75939: Status 404 returned error can't find the container with id 20590919bf7b0abe73aa3b715acf6dd2ada4acbe612d8147b956e78ae0c75939 Jul 11 03:43:33.731711 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:33.731656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nn9nn" event={"ID":"4a39043c-ddee-4dbc-94b8-d3e387668742","Type":"ContainerStarted","Data":"20590919bf7b0abe73aa3b715acf6dd2ada4acbe612d8147b956e78ae0c75939"} Jul 11 03:43:36.741087 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:36.741045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nn9nn" event={"ID":"4a39043c-ddee-4dbc-94b8-d3e387668742","Type":"ContainerStarted","Data":"fbe3c48e27f5e21cefd4089ab216fb2bf3d249df2f11dd6fae06adf3b5fda859"} Jul 11 03:43:36.741574 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:36.741101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nn9nn" event={"ID":"4a39043c-ddee-4dbc-94b8-d3e387668742","Type":"ContainerStarted","Data":"5e13998c3a2ab2fa25e3ef368d2de28c6dd7d38a0816be3e2eef12f0dcb0e306"} Jul 11 03:43:36.741574 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:36.741193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:36.756972 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:36.756921 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nn9nn" podStartSLOduration=17.457437554 podStartE2EDuration="19.756905321s" totalImagesPullingTime="2.299467767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:36.756227225 +0000 UTC m=+54.829843507" watchObservedRunningTime="2026-07-11 03:43:36.756905321 +0000 UTC m=+54.830521558" Jul 11 03:43:40.507034 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.507000 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx"] Jul 11 03:43:40.510533 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.510514 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-vsqjk"] Jul 11 03:43:40.510657 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.510639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.512791 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.512770 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-784c4f88b9-bfjjx"] Jul 11 03:43:40.512922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.512885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.515011 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.514992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.515441 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.515419 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.523742 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.523720 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 11 03:43:40.524805 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.524499 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.524805 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.524505 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rrwjm\"" Jul 11 03:43:40.525961 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.525938 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 11 03:43:40.526548 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.526525 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-whsx9\"" Jul 11 03:43:40.526807 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.526772 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.526900 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.526824 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.527076 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.527058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 11 03:43:40.528325 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.528301 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 11 03:43:40.529113 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.529076 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 11 03:43:40.529198 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.529158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 11 03:43:40.529382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.529367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nr2fs\"" Jul 11 03:43:40.529568 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.529549 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 11 03:43:40.544424 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.544397 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx"] Jul 11 03:43:40.552226 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.552203 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vsqjk"] Jul 11 03:43:40.555052 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.555031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 11 03:43:40.558869 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.558849 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 11 03:43:40.576399 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.576364 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-784c4f88b9-bfjjx"] Jul 11 03:43:40.602595 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-serving-cert\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.602595 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80037351-380c-49a1-b45a-6bd79f190263-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.602841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-config\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.602841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.602841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.602841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80037351-380c-49a1-b45a-6bd79f190263-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.602841 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhjb\" (UniqueName: \"kubernetes.io/projected/80037351-380c-49a1-b45a-6bd79f190263-kube-api-access-rxhjb\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602911 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vmj\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602973 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-trusted-ca\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.603010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.602993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxsm\" (UniqueName: \"kubernetes.io/projected/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-kube-api-access-szxsm\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.603197 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.603013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603197 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.603037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.603532 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.603511 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d"] Jul 11 03:43:40.606238 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.606223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.607756 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.607734 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k4bjs"] Jul 11 03:43:40.608767 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.608740 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 11 03:43:40.608884 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.608817 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5zvl8\"" Jul 11 03:43:40.608935 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.608879 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.608935 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.608888 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 11 03:43:40.609021 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.608978 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.610129 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.610111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.612435 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.612394 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.612566 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.612546 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 11 03:43:40.612708 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.612587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9j2vj\"" Jul 11 03:43:40.612895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.612872 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.613012 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.612980 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr"] Jul 11 03:43:40.615783 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.615738 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d95b78bdb-mnxz2"] Jul 11 03:43:40.615879 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.615840 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.618277 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.618260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.619791 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.619760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 11 03:43:40.621179 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.621151 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 11 03:43:40.624202 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.624179 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 11 03:43:40.624312 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.624204 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 11 03:43:40.624312 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.624265 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 11 03:43:40.624419 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.624371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 11 03:43:40.625171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.625152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5r9mh\"" Jul 11 03:43:40.625372 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.625201 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 11 03:43:40.625488 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.625213 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.625589 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.625239 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.625795 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.625779 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d"] Jul 11 03:43:40.628718 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.628695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k4bjs"] Jul 11 03:43:40.629453 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.629436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr"] Jul 11 03:43:40.644283 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.644253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d95b78bdb-mnxz2"] Jul 11 03:43:40.651212 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.651183 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-784c4f88b9-bfjjx"] Jul 11 03:43:40.651465 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:40.651422 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-l7vmj registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" podUID="63d3c51b-0913-4b94-ac15-7476926771f5" Jul 11 03:43:40.696289 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.696260 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt"] Jul 11 03:43:40.699081 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.699065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.701258 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.701233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b6rg9\"" Jul 11 03:43:40.701565 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.701548 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 11 03:43:40.701735 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.701716 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 11 03:43:40.701832 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.701737 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 11 03:43:40.701898 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.701883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 11 03:43:40.703433 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clxq\" (UniqueName: \"kubernetes.io/projected/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-kube-api-access-9clxq\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.703528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef37e3bf-38af-4470-87b2-75405cd63f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.703528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db85k\" (UniqueName: \"kubernetes.io/projected/ef37e3bf-38af-4470-87b2-75405cd63f5b-kube-api-access-db85k\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.703528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2md\" (UniqueName: \"kubernetes.io/projected/45127447-2a0a-4736-82e3-da2f9aa5b302-kube-api-access-nm2md\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.703528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80037351-380c-49a1-b45a-6bd79f190263-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhjb\" (UniqueName: \"kubernetes.io/projected/80037351-380c-49a1-b45a-6bd79f190263-kube-api-access-rxhjb\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-default-certificate\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7acdf46-b4ea-4f10-8c10-eea901a81a52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.703745 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-snapshots\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-serving-cert\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8k2\" (UniqueName: \"kubernetes.io/projected/6e4241b4-5633-4ddc-9de4-6873af06759a-kube-api-access-td8k2\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-metrics-certs\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.703913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef37e3bf-38af-4470-87b2-75405cd63f5b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.704040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-tmp\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.704364 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.704364 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.704364 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.704364 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vmj\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.704558 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.704617 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-trusted-ca\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxsm\" (UniqueName: \"kubernetes.io/projected/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-kube-api-access-szxsm\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80037351-380c-49a1-b45a-6bd79f190263-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80037351-380c-49a1-b45a-6bd79f190263-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7t7\" (UniqueName: \"kubernetes.io/projected/c7acdf46-b4ea-4f10-8c10-eea901a81a52-kube-api-access-mj7t7\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.704973 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.704873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4241b4-5633-4ddc-9de4-6873af06759a-service-ca-bundle\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.705336 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.705336 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.705474 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-stats-auth\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-config\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-tmp\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.705755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45127447-2a0a-4736-82e3-da2f9aa5b302-serving-cert\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.706068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.706507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-trusted-ca\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.706706 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.706703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-config\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.707224 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.706910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.707224 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.707111 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.707224 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.707204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-serving-cert\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.707899 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.707882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80037351-380c-49a1-b45a-6bd79f190263-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.708972 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.708950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.709781 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.709757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt"] Jul 11 03:43:40.717065 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.717038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.717473 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.717453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vmj\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj\") pod \"image-registry-784c4f88b9-bfjjx\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.717757 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.717738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhjb\" (UniqueName: \"kubernetes.io/projected/80037351-380c-49a1-b45a-6bd79f190263-kube-api-access-rxhjb\") pod \"cluster-monitoring-operator-5f7f494dd4-9fhlx\" (UID: \"80037351-380c-49a1-b45a-6bd79f190263\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.717835 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.717812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxsm\" (UniqueName: \"kubernetes.io/projected/7a63cb42-0f1c-49bb-926c-79db9f2ab0cd-kube-api-access-szxsm\") pod \"console-operator-7866497494-vsqjk\" (UID: \"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd\") " pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.750802 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.750771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.758280 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.758206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:40.807374 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807338 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807582 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807420 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807582 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807454 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807582 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807486 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807582 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807526 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7vmj\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807582 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807559 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807848 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807848 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807611 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls\") pod \"63d3c51b-0913-4b94-ac15-7476926771f5\" (UID: \"63d3c51b-0913-4b94-ac15-7476926771f5\") " Jul 11 03:43:40.807848 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.807848 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7t7\" (UniqueName: \"kubernetes.io/projected/c7acdf46-b4ea-4f10-8c10-eea901a81a52-kube-api-access-mj7t7\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4241b4-5633-4ddc-9de4-6873af06759a-service-ca-bundle\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-stats-auth\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.807978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-tmp\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45127447-2a0a-4736-82e3-da2f9aa5b302-serving-cert\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clxq\" (UniqueName: \"kubernetes.io/projected/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-kube-api-access-9clxq\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.808070 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef37e3bf-38af-4470-87b2-75405cd63f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808110 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db85k\" (UniqueName: \"kubernetes.io/projected/ef37e3bf-38af-4470-87b2-75405cd63f5b-kube-api-access-db85k\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2md\" (UniqueName: \"kubernetes.io/projected/45127447-2a0a-4736-82e3-da2f9aa5b302-kube-api-access-nm2md\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-default-certificate\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7acdf46-b4ea-4f10-8c10-eea901a81a52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-snapshots\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8k2\" (UniqueName: \"kubernetes.io/projected/6e4241b4-5633-4ddc-9de4-6873af06759a-kube-api-access-td8k2\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.808422 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-metrics-certs\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.808956 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef37e3bf-38af-4470-87b2-75405cd63f5b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.808956 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-tmp\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.808956 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-tmp\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.809435 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808024 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 03:43:40.809641 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:43:40.809795 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.808049 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:43:40.809932 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.809897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.810297 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.809497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-snapshots\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.810698 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.810545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45127447-2a0a-4736-82e3-da2f9aa5b302-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.810698 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.810584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef37e3bf-38af-4470-87b2-75405cd63f5b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.810698 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.810624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 03:43:40.810917 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.810711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4241b4-5633-4ddc-9de4-6873af06759a-service-ca-bundle\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.811232 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.811070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45127447-2a0a-4736-82e3-da2f9aa5b302-tmp\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.812190 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.812159 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:43:40.812344 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.812307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45127447-2a0a-4736-82e3-da2f9aa5b302-serving-cert\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.812414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.812378 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 03:43:40.813062 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.813025 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:43:40.813559 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.813515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-default-certificate\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.813660 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.813594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj" (OuterVolumeSpecName: "kube-api-access-l7vmj") pod "63d3c51b-0913-4b94-ac15-7476926771f5" (UID: "63d3c51b-0913-4b94-ac15-7476926771f5"). InnerVolumeSpecName "kube-api-access-l7vmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 03:43:40.813877 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.813855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7acdf46-b4ea-4f10-8c10-eea901a81a52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.813949 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.813874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef37e3bf-38af-4470-87b2-75405cd63f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.814431 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.814412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-stats-auth\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.814605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.814586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e4241b4-5633-4ddc-9de4-6873af06759a-metrics-certs\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.814649 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.814637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.817919 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.817898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8k2\" (UniqueName: \"kubernetes.io/projected/6e4241b4-5633-4ddc-9de4-6873af06759a-kube-api-access-td8k2\") pod \"router-default-6d95b78bdb-mnxz2\" (UID: \"6e4241b4-5633-4ddc-9de4-6873af06759a\") " pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.818633 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.818613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7t7\" (UniqueName: \"kubernetes.io/projected/c7acdf46-b4ea-4f10-8c10-eea901a81a52-kube-api-access-mj7t7\") pod \"managed-serviceaccount-addon-agent-76dccf9b58-6tc9d\" (UID: \"c7acdf46-b4ea-4f10-8c10-eea901a81a52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.820282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.820261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" Jul 11 03:43:40.820373 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.820342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2md\" (UniqueName: \"kubernetes.io/projected/45127447-2a0a-4736-82e3-da2f9aa5b302-kube-api-access-nm2md\") pod \"insights-operator-68d9dd94cf-k4bjs\" (UID: \"45127447-2a0a-4736-82e3-da2f9aa5b302\") " pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.820530 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.820514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db85k\" (UniqueName: \"kubernetes.io/projected/ef37e3bf-38af-4470-87b2-75405cd63f5b-kube-api-access-db85k\") pod \"kube-storage-version-migrator-operator-799cb7f958-qm9mt\" (UID: \"ef37e3bf-38af-4470-87b2-75405cd63f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:40.820875 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.820860 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clxq\" (UniqueName: \"kubernetes.io/projected/49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa-kube-api-access-9clxq\") pod \"klusterlet-addon-workmgr-5cc7647c98-rf8hr\" (UID: \"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.828414 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.828398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910187 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-bound-sa-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910226 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-trusted-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910240 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63d3c51b-0913-4b94-ac15-7476926771f5-ca-trust-extracted\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910254 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7vmj\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-kube-api-access-l7vmj\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910270 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-installation-pull-secrets\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910283 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63d3c51b-0913-4b94-ac15-7476926771f5-registry-certificates\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910301 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910295 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63d3c51b-0913-4b94-ac15-7476926771f5-registry-tls\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.910666 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.910308 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63d3c51b-0913-4b94-ac15-7476926771f5-image-registry-private-configuration\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:43:40.928983 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.928950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" Jul 11 03:43:40.935943 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.935915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" Jul 11 03:43:40.948527 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.943566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:40.950030 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.949993 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:40.962033 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.961988 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx"] Jul 11 03:43:40.998877 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:40.995574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vsqjk"] Jul 11 03:43:41.031109 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.030018 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" Jul 11 03:43:41.110355 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.110265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d"] Jul 11 03:43:41.134345 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.134225 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k4bjs"] Jul 11 03:43:41.139226 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:41.139191 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45127447_2a0a_4736_82e3_da2f9aa5b302.slice/crio-8a8eb9963f064696cc9fe5bc95185c011b60de2fbe0a4da781e15e1d4829c059 WatchSource:0}: Error finding container 8a8eb9963f064696cc9fe5bc95185c011b60de2fbe0a4da781e15e1d4829c059: Status 404 returned error can't find the container with id 8a8eb9963f064696cc9fe5bc95185c011b60de2fbe0a4da781e15e1d4829c059 Jul 11 03:43:41.150027 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.149999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr"] Jul 11 03:43:41.153661 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:41.153619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ffc6b7_2703_4fa2_82b5_0f81ff1b28fa.slice/crio-d04245ae8c29c1b5cfccf0dfe7a958280781e43d79f6cd0dc210df3b49cfa38e WatchSource:0}: Error finding container d04245ae8c29c1b5cfccf0dfe7a958280781e43d79f6cd0dc210df3b49cfa38e: Status 404 returned error can't find the container with id d04245ae8c29c1b5cfccf0dfe7a958280781e43d79f6cd0dc210df3b49cfa38e Jul 11 03:43:41.167023 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.166993 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d95b78bdb-mnxz2"] Jul 11 03:43:41.170917 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:41.170892 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4241b4_5633_4ddc_9de4_6873af06759a.slice/crio-c30d2f158214052a4901f498b0ff23f956dc816cd707961f4e41f1fba7118f16 WatchSource:0}: Error finding container c30d2f158214052a4901f498b0ff23f956dc816cd707961f4e41f1fba7118f16: Status 404 returned error can't find the container with id c30d2f158214052a4901f498b0ff23f956dc816cd707961f4e41f1fba7118f16 Jul 11 03:43:41.194513 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.194479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt"] Jul 11 03:43:41.201693 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:41.201640 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef37e3bf_38af_4470_87b2_75405cd63f5b.slice/crio-3b9948d5bbac41e1287f790f90c62acc090026481e45f8b5d44e928228af07ac WatchSource:0}: Error finding container 3b9948d5bbac41e1287f790f90c62acc090026481e45f8b5d44e928228af07ac: Status 404 returned error can't find the container with id 3b9948d5bbac41e1287f790f90c62acc090026481e45f8b5d44e928228af07ac Jul 11 03:43:41.757584 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.757545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" event={"ID":"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa","Type":"ContainerStarted","Data":"d04245ae8c29c1b5cfccf0dfe7a958280781e43d79f6cd0dc210df3b49cfa38e"} Jul 11 03:43:41.761300 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.760365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" event={"ID":"6e4241b4-5633-4ddc-9de4-6873af06759a","Type":"ContainerStarted","Data":"e73b6b9f61c6b0e1964f40825f74bf81b2161755ea4a5e07aa520b5206ae0f06"} Jul 11 03:43:41.761300 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.760419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" event={"ID":"6e4241b4-5633-4ddc-9de4-6873af06759a","Type":"ContainerStarted","Data":"c30d2f158214052a4901f498b0ff23f956dc816cd707961f4e41f1fba7118f16"} Jul 11 03:43:41.763624 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.763559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" event={"ID":"80037351-380c-49a1-b45a-6bd79f190263","Type":"ContainerStarted","Data":"4d27d125ade69085dd2c3798ef269d3a5d0e9781099859e963e0afbb353cb1e7"} Jul 11 03:43:41.765961 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.765881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" event={"ID":"ef37e3bf-38af-4470-87b2-75405cd63f5b","Type":"ContainerStarted","Data":"3b9948d5bbac41e1287f790f90c62acc090026481e45f8b5d44e928228af07ac"} Jul 11 03:43:41.768000 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.767968 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" event={"ID":"45127447-2a0a-4736-82e3-da2f9aa5b302","Type":"ContainerStarted","Data":"8a8eb9963f064696cc9fe5bc95185c011b60de2fbe0a4da781e15e1d4829c059"} Jul 11 03:43:41.770015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.769951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vsqjk" event={"ID":"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd","Type":"ContainerStarted","Data":"23427672a6640518e467bf3707dc6d6b8bc4747de810d66d805b76dd76fb4c85"} Jul 11 03:43:41.771402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.771383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784c4f88b9-bfjjx" Jul 11 03:43:41.775637 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.775543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" event={"ID":"c7acdf46-b4ea-4f10-8c10-eea901a81a52","Type":"ContainerStarted","Data":"e91b7044c1e2025f4d792485d05cfa19096d84374caf2983d6867c6235b4fdc7"} Jul 11 03:43:41.804342 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.804215 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" podStartSLOduration=1.804192237 podStartE2EDuration="1.804192237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:41.783521841 +0000 UTC m=+59.857138104" watchObservedRunningTime="2026-07-11 03:43:41.804192237 +0000 UTC m=+59.877808477" Jul 11 03:43:41.813332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.813299 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-784c4f88b9-bfjjx"] Jul 11 03:43:41.815072 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.814983 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-784c4f88b9-bfjjx"] Jul 11 03:43:41.950631 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.950598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:41.953839 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:41.953604 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:42.430415 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:42.430383 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d3c51b-0913-4b94-ac15-7476926771f5" path="/var/lib/kubelet/pods/63d3c51b-0913-4b94-ac15-7476926771f5/volumes" Jul 11 03:43:42.699144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:42.699027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz6zf" Jul 11 03:43:42.794216 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:42.794041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:42.796648 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:42.796405 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d95b78bdb-mnxz2" Jul 11 03:43:46.746455 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:46.746379 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nn9nn" Jul 11 03:43:47.283817 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.283778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:47.283999 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.283829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:47.284076 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.284047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:47.286256 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.286233 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 11 03:43:47.286950 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.286932 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 11 03:43:47.287037 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.286963 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 11 03:43:47.296474 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.296448 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 11 03:43:47.297270 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.297246 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2de47295-6cd6-4b77-a057-6d8f5d1b22bd-original-pull-secret\") pod \"global-pull-secret-syncer-bz6lb\" (UID: \"2de47295-6cd6-4b77-a057-6d8f5d1b22bd\") " pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:47.297404 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.297389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9dec5b8-a00b-4113-9823-1bf997999428-metrics-certs\") pod \"network-metrics-daemon-2qmkx\" (UID: \"e9dec5b8-a00b-4113-9823-1bf997999428\") " pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:47.308047 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.308015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llln\" (UniqueName: \"kubernetes.io/projected/34cf96ad-9684-4763-a44e-541ab12906e7-kube-api-access-6llln\") pod \"network-check-target-rqclv\" (UID: \"34cf96ad-9684-4763-a44e-541ab12906e7\") " pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:47.432151 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.432115 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz6lb" Jul 11 03:43:47.538428 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.538349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6d8m4\"" Jul 11 03:43:47.545215 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.545184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-57qws\"" Jul 11 03:43:47.546184 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.546156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2qmkx" Jul 11 03:43:47.554120 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:47.554084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:48.286510 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.286380 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz6lb"] Jul 11 03:43:48.314281 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.314050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rqclv"] Jul 11 03:43:48.325616 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.325353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2qmkx"] Jul 11 03:43:48.824198 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.824093 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" event={"ID":"49ffc6b7-2703-4fa2-82b5-0f81ff1b28fa","Type":"ContainerStarted","Data":"54fe1e5516c8b0b93a07d45f20003d62d1853504ac2c71ea503487499f1ad041"} Jul 11 03:43:48.826109 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.824536 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:48.826834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.826741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" Jul 11 03:43:48.829807 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.828256 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" event={"ID":"80037351-380c-49a1-b45a-6bd79f190263","Type":"ContainerStarted","Data":"19589582f16aa14ed6a262b43d4880cec9451e5462c269d39fc4b03b58c92914"} Jul 11 03:43:48.833522 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.833442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bz6lb" event={"ID":"2de47295-6cd6-4b77-a057-6d8f5d1b22bd","Type":"ContainerStarted","Data":"86aa63d295029ed8fc2b7949e1ce56b78cb8ab8a811ac1d245ab3c635f9902da"} Jul 11 03:43:48.835406 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.835328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2qmkx" event={"ID":"e9dec5b8-a00b-4113-9823-1bf997999428","Type":"ContainerStarted","Data":"a44ec7e840c892dec0ab8f788cb045c1ad40e9c3059bfe784cc5e4af69f75e2e"} Jul 11 03:43:48.838282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.837746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" event={"ID":"ef37e3bf-38af-4470-87b2-75405cd63f5b","Type":"ContainerStarted","Data":"ebe664eb7fbbce9d1d5b8ce48d9d3c32063d973fc90b37b0210aa6fad480160a"} Jul 11 03:43:48.840860 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.840350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" event={"ID":"45127447-2a0a-4736-82e3-da2f9aa5b302","Type":"ContainerStarted","Data":"9db8e55dd70ce2eae372c390b5458c5a3b7a53790f9c8087cf7fd9f17a20c4ba"} Jul 11 03:43:48.846107 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.845894 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc7647c98-rf8hr" podStartSLOduration=1.929882485 podStartE2EDuration="8.845875418s" totalImagesPullingTime="6.915992933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.844381064 +0000 UTC m=+66.917997314" watchObservedRunningTime="2026-07-11 03:43:48.845875418 +0000 UTC m=+66.919491672" Jul 11 03:43:48.851774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.848257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vsqjk" event={"ID":"7a63cb42-0f1c-49bb-926c-79db9f2ab0cd","Type":"ContainerStarted","Data":"3f023cab4eac152303aea8f53836797d247884ef0df93571de5bf6509b65c558"} Jul 11 03:43:48.851774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.850191 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:48.852806 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.852732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rqclv" event={"ID":"34cf96ad-9684-4763-a44e-541ab12906e7","Type":"ContainerStarted","Data":"494a7b01b5b346a42516736222245ee58caa27d7da3b0ec4006161477b81025f"} Jul 11 03:43:48.856128 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.855641 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" event={"ID":"c7acdf46-b4ea-4f10-8c10-eea901a81a52","Type":"ContainerStarted","Data":"7023c9f3c0422f1525acef84178591833802161fec292c2042e2eaee0a82b7a7"} Jul 11 03:43:48.862611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.858586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-vsqjk" Jul 11 03:43:48.862611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.862265 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9fhlx" podStartSLOduration=1.7828773359999999 podStartE2EDuration="8.862248857s" totalImagesPullingTime="7.079371521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.860995338 +0000 UTC m=+66.934611599" watchObservedRunningTime="2026-07-11 03:43:48.862248857 +0000 UTC m=+66.935865117" Jul 11 03:43:48.893052 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.892996 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-qm9mt" podStartSLOduration=2.040419381 podStartE2EDuration="8.892977983s" totalImagesPullingTime="6.852558602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.892665067 +0000 UTC m=+66.966281327" watchObservedRunningTime="2026-07-11 03:43:48.892977983 +0000 UTC m=+66.966594231" Jul 11 03:43:48.913718 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.907937 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-k4bjs" podStartSLOduration=1.992644945 podStartE2EDuration="8.907917086s" totalImagesPullingTime="6.915272141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.907166789 +0000 UTC m=+66.980783049" watchObservedRunningTime="2026-07-11 03:43:48.907917086 +0000 UTC m=+66.981533347" Jul 11 03:43:48.954066 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.954002 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76dccf9b58-6tc9d" podStartSLOduration=2.014353941 podStartE2EDuration="8.953977785s" totalImagesPullingTime="6.939623844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.930258124 +0000 UTC m=+67.003874384" watchObservedRunningTime="2026-07-11 03:43:48.953977785 +0000 UTC m=+67.027594042" Jul 11 03:43:48.958181 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:48.954439 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-vsqjk" podStartSLOduration=1.912613616 podStartE2EDuration="8.954428981s" totalImagesPullingTime="7.041815365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:48.953083973 +0000 UTC m=+67.026700235" watchObservedRunningTime="2026-07-11 03:43:48.954428981 +0000 UTC m=+67.028045242" Jul 11 03:43:49.377703 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.377519 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrtn"] Jul 11 03:43:49.389906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.385956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.389906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.388621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrtn"] Jul 11 03:43:49.390125 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.389997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ddl6z\"" Jul 11 03:43:49.390255 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.390234 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 11 03:43:49.390332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.389997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 11 03:43:49.515751 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.515709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.515922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.515770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b491ff8-61f0-426e-adfd-c105a593867b-crio-socket\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.515922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.515800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jss2\" (UniqueName: \"kubernetes.io/projected/5b491ff8-61f0-426e-adfd-c105a593867b-kube-api-access-6jss2\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.515922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.515842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b491ff8-61f0-426e-adfd-c105a593867b-data-volume\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.516023 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.515931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.616737 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.616698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.616936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.616770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.616936 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:49.616848 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 03:43:49.616936 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:49.616889 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 03:43:49.616936 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:49.616930 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:50.116908571 +0000 UTC m=+68.190524824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : configmap "kube-rbac-proxy" not found Jul 11 03:43:49.617145 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:49.616957 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:50.11693941 +0000 UTC m=+68.190555678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : secret "insights-runtime-extractor-tls" not found Jul 11 03:43:49.617145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.616983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b491ff8-61f0-426e-adfd-c105a593867b-crio-socket\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.617145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.617034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jss2\" (UniqueName: \"kubernetes.io/projected/5b491ff8-61f0-426e-adfd-c105a593867b-kube-api-access-6jss2\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.617145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.617120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b491ff8-61f0-426e-adfd-c105a593867b-data-volume\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.617347 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.617198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b491ff8-61f0-426e-adfd-c105a593867b-crio-socket\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.617724 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.617664 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b491ff8-61f0-426e-adfd-c105a593867b-data-volume\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.627760 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.627700 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jss2\" (UniqueName: \"kubernetes.io/projected/5b491ff8-61f0-426e-adfd-c105a593867b-kube-api-access-6jss2\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:49.865858 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:49.865822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2qmkx" event={"ID":"e9dec5b8-a00b-4113-9823-1bf997999428","Type":"ContainerStarted","Data":"ea373170fbbceca067d7b91a0bef1ea68c113e864f6bd2e4774a0a647fcca2b5"} Jul 11 03:43:50.122807 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:50.122769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:50.122981 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:50.122832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:50.122981 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:50.122965 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 03:43:50.123081 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:50.123018 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 03:43:50.123123 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:50.123030 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:51.123013131 +0000 UTC m=+69.196629389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : configmap "kube-rbac-proxy" not found Jul 11 03:43:50.123123 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:50.123111 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:51.123096542 +0000 UTC m=+69.196712785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : secret "insights-runtime-extractor-tls" not found Jul 11 03:43:50.826049 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:50.826015 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nn9nn_4a39043c-ddee-4dbc-94b8-d3e387668742/dns/0.log" Jul 11 03:43:50.873832 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:50.873743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2qmkx" event={"ID":"e9dec5b8-a00b-4113-9823-1bf997999428","Type":"ContainerStarted","Data":"6d9139baf48ce7e40e9fa5c883dcc9701b167a2ad5e646cc7107deafe5b3e226"} Jul 11 03:43:50.887812 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:50.887751 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2qmkx" podStartSLOduration=67.538174744 podStartE2EDuration="1m8.887734554s" totalImagesPullingTime="1.34955981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:50.887363244 +0000 UTC m=+68.960979526" watchObservedRunningTime="2026-07-11 03:43:50.887734554 +0000 UTC m=+68.961350856" Jul 11 03:43:51.000642 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:51.000611 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nn9nn_4a39043c-ddee-4dbc-94b8-d3e387668742/kube-rbac-proxy/0.log" Jul 11 03:43:51.132743 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:51.132702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:51.132932 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:51.132770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:51.132932 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:51.132881 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 03:43:51.132932 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:51.132902 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 03:43:51.133091 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:51.132961 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:53.132939586 +0000 UTC m=+71.206555841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : secret "insights-runtime-extractor-tls" not found Jul 11 03:43:51.133091 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:51.132981 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:53.132972246 +0000 UTC m=+71.206588489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : configmap "kube-rbac-proxy" not found Jul 11 03:43:51.999875 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:51.999847 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bn8z_c67a41b4-1865-4ca4-a40c-4bfe16e517b6/dns-node-resolver/0.log" Jul 11 03:43:53.152845 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.152800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:53.153391 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.152857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:53.153391 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:53.152974 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 03:43:53.153391 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:53.153058 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:57.153035481 +0000 UTC m=+75.226651732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : secret "insights-runtime-extractor-tls" not found Jul 11 03:43:53.153391 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:53.152976 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 03:43:53.153391 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:53.153154 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:43:57.153134398 +0000 UTC m=+75.226750634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : configmap "kube-rbac-proxy" not found Jul 11 03:43:53.400044 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.400008 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6v5k_5eb7690c-2a06-447c-8054-917b81c5ae3d/node-ca/0.log" Jul 11 03:43:53.600431 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.600365 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d95b78bdb-mnxz2_6e4241b4-5633-4ddc-9de4-6873af06759a/router/0.log" Jul 11 03:43:53.886528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.886482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bz6lb" event={"ID":"2de47295-6cd6-4b77-a057-6d8f5d1b22bd","Type":"ContainerStarted","Data":"922dcc4b3a13d4248b54745d47a005ad2a1bbfe6fc3c261726130962f784fee7"} Jul 11 03:43:53.888090 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.888065 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rqclv" event={"ID":"34cf96ad-9684-4763-a44e-541ab12906e7","Type":"ContainerStarted","Data":"9a44d7ebb33788723c6d48df07af24bf2b20039cd7f5fa2865948de002cdeb5b"} Jul 11 03:43:53.888215 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.888193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:43:53.905253 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.905194 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bz6lb" podStartSLOduration=66.479446995 podStartE2EDuration="1m11.905175703s" totalImagesPullingTime="5.425728708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:53.903917443 +0000 UTC m=+71.977533729" watchObservedRunningTime="2026-07-11 03:43:53.905175703 +0000 UTC m=+71.978791963" Jul 11 03:43:53.918000 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:53.917931 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rqclv" podStartSLOduration=66.528555789 podStartE2EDuration="1m11.917915502s" totalImagesPullingTime="5.389359713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:43:53.917276169 +0000 UTC m=+71.990892427" watchObservedRunningTime="2026-07-11 03:43:53.917915502 +0000 UTC m=+71.991531759" Jul 11 03:43:54.001622 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:54.001587 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j5h5d_1d17d352-eb89-4c0e-8b78-71d76d5b567d/serve-healthcheck-canary/0.log" Jul 11 03:43:54.801817 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:54.801786 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-qm9mt_ef37e3bf-38af-4470-87b2-75405cd63f5b/kube-storage-version-migrator-operator/0.log" Jul 11 03:43:57.189136 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:57.189096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:57.189555 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:57.189144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:43:57.189555 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:57.189234 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 03:43:57.189555 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:57.189267 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 03:43:57.189555 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:57.189291 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:44:05.189276483 +0000 UTC m=+83.262892720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : configmap "kube-rbac-proxy" not found Jul 11 03:43:57.189555 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:43:57.189340 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls podName:5b491ff8-61f0-426e-adfd-c105a593867b nodeName:}" failed. No retries permitted until 2026-07-11 03:44:05.189325893 +0000 UTC m=+83.262942130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhrtn" (UID: "5b491ff8-61f0-426e-adfd-c105a593867b") : secret "insights-runtime-extractor-tls" not found Jul 11 03:43:58.189829 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.189798 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q4bdb"] Jul 11 03:43:58.193310 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.193287 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.195562 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.195529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 11 03:43:58.195713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.195585 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 11 03:43:58.195713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.195620 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mqg6h\"" Jul 11 03:43:58.195878 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.195862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 11 03:43:58.196887 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.196872 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 11 03:43:58.296813 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.296775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.296828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-metrics-client-ca\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.296863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zngl\" (UniqueName: \"kubernetes.io/projected/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-kube-api-access-5zngl\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.296923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-root\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.296964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-tls\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.297006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-wtmp\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297246 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.297034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297246 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.297064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-sys\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.297246 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.297085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-textfile\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398367 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398377 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-metrics-client-ca\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zngl\" (UniqueName: \"kubernetes.io/projected/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-kube-api-access-5zngl\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-root\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-tls\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-wtmp\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-sys\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-textfile\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398941 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-sys\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398941 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-root\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.398941 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.398779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-wtmp\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.399080 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.399040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-textfile\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.399140 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.399076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.399140 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.399089 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-metrics-client-ca\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.401239 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.401211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.401359 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.401328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-node-exporter-tls\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.405910 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.405886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zngl\" (UniqueName: \"kubernetes.io/projected/9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4-kube-api-access-5zngl\") pod \"node-exporter-q4bdb\" (UID: \"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4\") " pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.504071 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.503989 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4bdb" Jul 11 03:43:58.516251 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:43:58.516208 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf33ff5_d1b2_4ec5_a3f7_6d2d8ca97df4.slice/crio-3fd17011e9628b7ddf04ef51cf55076c91267b4cac1d6d231eb082d340bce5f3 WatchSource:0}: Error finding container 3fd17011e9628b7ddf04ef51cf55076c91267b4cac1d6d231eb082d340bce5f3: Status 404 returned error can't find the container with id 3fd17011e9628b7ddf04ef51cf55076c91267b4cac1d6d231eb082d340bce5f3 Jul 11 03:43:58.905347 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:58.905300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4bdb" event={"ID":"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4","Type":"ContainerStarted","Data":"3fd17011e9628b7ddf04ef51cf55076c91267b4cac1d6d231eb082d340bce5f3"} Jul 11 03:43:59.910204 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:59.910170 2565 generic.go:356] "Generic (PLEG): container finished" podID="9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4" containerID="1f7ec4ad8e75d24efe3bac3ac61e026e3f20620d51b7550e0c35116d8e205de5" exitCode=0 Jul 11 03:43:59.910588 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:43:59.910210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4bdb" event={"ID":"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4","Type":"ContainerDied","Data":"1f7ec4ad8e75d24efe3bac3ac61e026e3f20620d51b7550e0c35116d8e205de5"} Jul 11 03:44:00.915277 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:00.915240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4bdb" event={"ID":"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4","Type":"ContainerStarted","Data":"7ce8c5089fabf2241735abc6d92cb0035633a2b053cdf88eca724280f047a05a"} Jul 11 03:44:00.915645 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:00.915295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4bdb" event={"ID":"9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4","Type":"ContainerStarted","Data":"495fb10140b3f609ac19da53269a386088a6625559fa176590e0871731f5043a"} Jul 11 03:44:00.934993 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:00.934932 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q4bdb" podStartSLOduration=1.848787915 podStartE2EDuration="2.934916443s" totalImagesPullingTime="1.013762122s" totalInitContainerRuntime="72.366406ms" isStatefulPod=true podCreationTimestamp="2026-07-11 03:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:00.93341478 +0000 UTC m=+79.007031038" watchObservedRunningTime="2026-07-11 03:44:00.934916443 +0000 UTC m=+79.008532701" Jul 11 03:44:02.551360 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.551326 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57b8fd6656-dfbc2"] Jul 11 03:44:02.554453 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.554432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.557565 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557533 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 11 03:44:02.557718 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557560 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1j3dd3ufqh23\"" Jul 11 03:44:02.557718 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557670 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dm8hn\"" Jul 11 03:44:02.557834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 11 03:44:02.557834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557567 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 11 03:44:02.557834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.557561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 11 03:44:02.563498 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.563475 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57b8fd6656-dfbc2"] Jul 11 03:44:02.632353 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-tls\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632506 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-client-certs\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632506 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-metrics-server-audit-profiles\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632506 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632461 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a45c6c12-eca0-4016-9e1a-be29f330841f-audit-log\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632506 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632657 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkm5\" (UniqueName: \"kubernetes.io/projected/a45c6c12-eca0-4016-9e1a-be29f330841f-kube-api-access-ktkm5\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.632657 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.632587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-client-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733732 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a45c6c12-eca0-4016-9e1a-be29f330841f-audit-log\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733883 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733883 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkm5\" (UniqueName: \"kubernetes.io/projected/a45c6c12-eca0-4016-9e1a-be29f330841f-kube-api-access-ktkm5\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733883 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-client-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733883 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-tls\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.733883 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-client-certs\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.734132 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.733893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-metrics-server-audit-profiles\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.734207 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.734185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a45c6c12-eca0-4016-9e1a-be29f330841f-audit-log\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.734552 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.734510 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.735102 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.735078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a45c6c12-eca0-4016-9e1a-be29f330841f-metrics-server-audit-profiles\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.736528 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.736504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-tls\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.736614 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.736582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-client-ca-bundle\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.736654 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.736606 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a45c6c12-eca0-4016-9e1a-be29f330841f-secret-metrics-server-client-certs\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.741250 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.741226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkm5\" (UniqueName: \"kubernetes.io/projected/a45c6c12-eca0-4016-9e1a-be29f330841f-kube-api-access-ktkm5\") pod \"metrics-server-57b8fd6656-dfbc2\" (UID: \"a45c6c12-eca0-4016-9e1a-be29f330841f\") " pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:02.864373 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:02.864292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:03.013484 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:03.013444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57b8fd6656-dfbc2"] Jul 11 03:44:03.016314 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:44:03.016283 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45c6c12_eca0_4016_9e1a_be29f330841f.slice/crio-c0d0add420a78a4f5cf39f112a8d82781dc14e19abcbf9ba09f8cbde570c5481 WatchSource:0}: Error finding container c0d0add420a78a4f5cf39f112a8d82781dc14e19abcbf9ba09f8cbde570c5481: Status 404 returned error can't find the container with id c0d0add420a78a4f5cf39f112a8d82781dc14e19abcbf9ba09f8cbde570c5481 Jul 11 03:44:03.928779 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:03.928738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" event={"ID":"a45c6c12-eca0-4016-9e1a-be29f330841f","Type":"ContainerStarted","Data":"c0d0add420a78a4f5cf39f112a8d82781dc14e19abcbf9ba09f8cbde570c5481"} Jul 11 03:44:04.397451 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.397409 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:44:04.402359 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.402331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.404531 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.404508 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 11 03:44:04.404695 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.404645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c9mw5\"" Jul 11 03:44:04.405015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.404930 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 11 03:44:04.405015 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.405001 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 11 03:44:04.405155 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.405027 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 11 03:44:04.406502 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.406471 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.407181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.407451 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.407793 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.407989 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6kc6388pii4va\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408533 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408595 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 11 03:44:04.409144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.408920 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 11 03:44:04.416433 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.416414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:44:04.549494 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549494 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549758 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549758 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549758 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549758 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549816 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.549916 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.549959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.550332 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.550250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wj2\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651314 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651634 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651648 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wj2\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.651895 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.651994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.652109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652655 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.652484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.652655 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.652515 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.654277 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.654254 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.654650 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.654624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.654769 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.654657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.654827 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.654809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.655535 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.655325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.655535 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.655416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.656432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.656161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.656432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.656182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.656432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.656354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.656777 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.656693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.657391 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.657176 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.657391 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.657276 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.657762 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.657740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.658111 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.658092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.659167 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.659144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.659642 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.659622 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wj2\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2\") pod \"prometheus-k8s-0\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.715574 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.715533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:04.854413 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.854379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:44:04.862400 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:44:04.860277 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf328fba0_fcb0_4671_94bc_b3a858340c80.slice/crio-4c4fa41b3fd7a7437b4d127bbed0467ac92c2d7e321a6d14bc705cbf8eb79732 WatchSource:0}: Error finding container 4c4fa41b3fd7a7437b4d127bbed0467ac92c2d7e321a6d14bc705cbf8eb79732: Status 404 returned error can't find the container with id 4c4fa41b3fd7a7437b4d127bbed0467ac92c2d7e321a6d14bc705cbf8eb79732 Jul 11 03:44:04.933553 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.933458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"4c4fa41b3fd7a7437b4d127bbed0467ac92c2d7e321a6d14bc705cbf8eb79732"} Jul 11 03:44:04.935033 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.935011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" event={"ID":"a45c6c12-eca0-4016-9e1a-be29f330841f","Type":"ContainerStarted","Data":"a5d3ed83846e558f42e591ff4b3b52b3179c32ce140885d82a67815ac18666ca"} Jul 11 03:44:04.969542 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:04.969411 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" podStartSLOduration=1.39726285 podStartE2EDuration="2.969390666s" totalImagesPullingTime="1.572127816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:04.969040497 +0000 UTC m=+83.042656757" watchObservedRunningTime="2026-07-11 03:44:04.969390666 +0000 UTC m=+83.043006928" Jul 11 03:44:05.258616 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.258522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:44:05.258616 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.258578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:44:05.259087 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.259065 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b491ff8-61f0-426e-adfd-c105a593867b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:44:05.260952 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.260933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b491ff8-61f0-426e-adfd-c105a593867b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhrtn\" (UID: \"5b491ff8-61f0-426e-adfd-c105a593867b\") " pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:44:05.301404 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.301363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhrtn" Jul 11 03:44:05.450770 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.450739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhrtn"] Jul 11 03:44:05.454399 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:44:05.454376 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b491ff8_61f0_426e_adfd_c105a593867b.slice/crio-f5a5ddadf5ce58e556f29f7d165f8770ea38279bf8aa3db626ceaa214575c710 WatchSource:0}: Error finding container f5a5ddadf5ce58e556f29f7d165f8770ea38279bf8aa3db626ceaa214575c710: Status 404 returned error can't find the container with id f5a5ddadf5ce58e556f29f7d165f8770ea38279bf8aa3db626ceaa214575c710 Jul 11 03:44:05.943481 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.943432 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} Jul 11 03:44:05.947171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:05.947112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrtn" event={"ID":"5b491ff8-61f0-426e-adfd-c105a593867b","Type":"ContainerStarted","Data":"f5a5ddadf5ce58e556f29f7d165f8770ea38279bf8aa3db626ceaa214575c710"} Jul 11 03:44:06.953443 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:06.953390 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" exitCode=0 Jul 11 03:44:06.953921 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:06.953459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} Jul 11 03:44:09.968556 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:09.968516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrtn" event={"ID":"5b491ff8-61f0-426e-adfd-c105a593867b","Type":"ContainerStarted","Data":"0fa5d325a4284e5402d07426c1b70feb9972d689cbfa667a09646379a693b88d"} Jul 11 03:44:10.975924 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:10.975890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrtn" event={"ID":"5b491ff8-61f0-426e-adfd-c105a593867b","Type":"ContainerStarted","Data":"89e8f7ea7115333e745df86e8fd38617efd648f1139b382ac8c4bb5cc8492a9c"} Jul 11 03:44:11.983541 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:11.983500 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} Jul 11 03:44:11.984001 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:11.983593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} Jul 11 03:44:14.997861 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:14.997825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhrtn" event={"ID":"5b491ff8-61f0-426e-adfd-c105a593867b","Type":"ContainerStarted","Data":"2fb19819e1b9470939b82e25567bbb498483febd46b33667d67bf78b1972c0e0"} Jul 11 03:44:15.001137 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.001108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} Jul 11 03:44:15.001276 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.001181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} Jul 11 03:44:15.001276 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.001222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} Jul 11 03:44:15.001276 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.001259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerStarted","Data":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} Jul 11 03:44:15.014568 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.014518 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zhrtn" podStartSLOduration=17.470636788 podStartE2EDuration="26.014502233s" totalImagesPullingTime="8.543865445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 03:43:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:15.013339787 +0000 UTC m=+93.086956081" watchObservedRunningTime="2026-07-11 03:44:15.014502233 +0000 UTC m=+93.088118491" Jul 11 03:44:15.037472 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:15.037400 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.993783508 podStartE2EDuration="11.037376502s" totalImagesPullingTime="8.008319144s" totalInitContainerRuntime="35.27385ms" isStatefulPod=false podCreationTimestamp="2026-07-11 03:44:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:44:15.036272443 +0000 UTC m=+93.109888703" watchObservedRunningTime="2026-07-11 03:44:15.037376502 +0000 UTC m=+93.110992762" Jul 11 03:44:19.715730 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:19.715694 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:44:22.864913 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:22.864871 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:22.865302 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:22.864975 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:24.894367 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:24.894335 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rqclv" Jul 11 03:44:42.870453 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:42.870423 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:44:42.874405 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:44:42.874380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57b8fd6656-dfbc2" Jul 11 03:45:04.716020 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:04.715981 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:04.732338 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:04.732312 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:05.201735 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:05.201710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:22.747284 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747248 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747527 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="prometheus" containerID="cri-o://2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" gracePeriod=600 Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747558 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="thanos-sidecar" containerID="cri-o://0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" gracePeriod=600 Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747558 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy" containerID="cri-o://a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" gracePeriod=600 Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747594 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-web" containerID="cri-o://2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" gracePeriod=600 Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747622 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="config-reloader" containerID="cri-o://3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" gracePeriod=600 Jul 11 03:45:22.747734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.747634 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" gracePeriod=600 Jul 11 03:45:22.996610 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:22.996582 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.123720 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123720 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123702 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123742 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123766 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.123936 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.123847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124169 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124005 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124169 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124169 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124169 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124154 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124220 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124473 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9wj2\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.124713 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124605 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0\") pod \"f328fba0-fcb0-4671-94bc-b3a858340c80\" (UID: \"f328fba0-fcb0-4671-94bc-b3a858340c80\") " Jul 11 03:45:23.125076 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124898 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.125076 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.124902 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.125262 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.125237 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.126443 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.126276 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.126443 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.126394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.126946 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.126717 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.127270 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.127231 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.127645 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.127613 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.127645 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.127621 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 03:45:23.127818 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.127740 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config" (OuterVolumeSpecName: "config") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.127946 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.127832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.128331 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.128297 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.128628 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.128608 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 03:45:23.128840 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.128809 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.128922 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.128899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.129734 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.129712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 03:45:23.129957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.129925 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2" (OuterVolumeSpecName: "kube-api-access-m9wj2") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "kube-api-access-m9wj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 03:45:23.129957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.129931 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.130098 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.129988 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out" (OuterVolumeSpecName: "config-out") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 03:45:23.138257 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.138232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config" (OuterVolumeSpecName: "web-config") pod "f328fba0-fcb0-4671-94bc-b3a858340c80" (UID: "f328fba0-fcb0-4671-94bc-b3a858340c80"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 03:45:23.225605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225564 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225597 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225609 2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-kube-rbac-proxy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225617 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225626 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225635 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-config-out\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225643 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225651 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225662 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225671 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-configmap-metrics-client-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225704 2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225714 2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-metrics-client-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225722 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-db\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225731 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-tls-assets\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225739 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wj2\" (UniqueName: \"kubernetes.io/projected/f328fba0-fcb0-4671-94bc-b3a858340c80-kube-api-access-m9wj2\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225747 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f328fba0-fcb0-4671-94bc-b3a858340c80-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225756 2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-secret-grpc-tls\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225764 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-web-config\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.225866 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.225771 2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f328fba0-fcb0-4671-94bc-b3a858340c80-config\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 03:45:23.257048 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257012 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" exitCode=0 Jul 11 03:45:23.257048 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257040 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" exitCode=0 Jul 11 03:45:23.257048 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257048 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" exitCode=0 Jul 11 03:45:23.257048 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257054 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" exitCode=0 Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257059 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" exitCode=0 Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257064 2565 generic.go:356] "Generic (PLEG): container finished" podID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" exitCode=0 Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257110 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} Jul 11 03:45:23.257282 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257270 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.257521 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} Jul 11 03:45:23.257521 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} Jul 11 03:45:23.257521 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} Jul 11 03:45:23.257521 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.257405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f328fba0-fcb0-4671-94bc-b3a858340c80","Type":"ContainerDied","Data":"4c4fa41b3fd7a7437b4d127bbed0467ac92c2d7e321a6d14bc705cbf8eb79732"} Jul 11 03:45:23.265448 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.265422 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.273218 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.273201 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.280054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.280031 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:23.280992 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.280975 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.284078 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.284055 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:23.288225 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.288207 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.295432 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.295412 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.302881 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.302862 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.309478 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.309463 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.309816 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.309796 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.309882 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.309824 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.309882 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.309866 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.310113 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.310096 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.310170 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310119 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.310170 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310134 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.310333 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.310317 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.310371 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310337 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.310371 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310349 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.310541 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.310523 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.310605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310552 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.310605 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310575 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.310818 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.310802 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.310874 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310822 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.310874 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.310839 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.311024 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.311009 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.311061 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311028 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.311061 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311044 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.311226 ip-10-0-138-240 kubenswrapper[2565]: E0711 03:45:23.311207 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.311264 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311235 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.311264 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311250 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.311448 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311433 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.311490 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311448 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.311610 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311595 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.311650 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311610 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.311820 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311805 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.311864 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311821 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.311994 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311979 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.312031 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.311995 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.312171 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312156 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.312212 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312172 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.312357 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312341 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.312400 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312359 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.312578 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312561 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.312645 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312580 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.312838 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312822 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.312890 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.312838 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.313025 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313011 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.313066 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313026 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.313214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313198 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.313254 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313214 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.313468 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313448 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.313520 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313469 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.313668 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313651 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.313737 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313669 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.313865 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313850 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.313905 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.313865 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.314042 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314026 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.314096 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314041 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.314206 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314191 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.314250 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314206 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.314386 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314368 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.314425 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314387 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.314564 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314548 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.314602 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314566 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.314774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314759 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.314829 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314774 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.314957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314942 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.314999 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.314957 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.315144 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315125 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.315210 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315148 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.315357 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315344 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.315401 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315357 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.315550 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315535 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.315597 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315550 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.315787 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315766 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.315847 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315789 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.316001 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.315983 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.316041 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316003 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.316228 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316213 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.316270 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316228 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.316444 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316419 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.316444 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316440 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.316669 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316651 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.316774 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316669 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.317010 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.316985 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.317106 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.317012 2565 scope.go:117] "RemoveContainer" containerID="4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34" Jul 11 03:45:23.317640 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.317565 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34"} err="failed to get container status \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": rpc error: code = NotFound desc = could not find container \"4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34\": container with ID starting with 4deb7d347493edcb4de487dc344aa4c0db6f7a5277c8611e2515a53a13309b34 not found: ID does not exist" Jul 11 03:45:23.317640 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.317590 2565 scope.go:117] "RemoveContainer" containerID="a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb" Jul 11 03:45:23.317891 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.317863 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb"} err="failed to get container status \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": rpc error: code = NotFound desc = could not find container \"a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb\": container with ID starting with a769958b658c4bc73e8b258d99f12d46896d0ce5357c76b674bddd28437c92bb not found: ID does not exist" Jul 11 03:45:23.317891 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.317891 2565 scope.go:117] "RemoveContainer" containerID="2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d" Jul 11 03:45:23.318137 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318111 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d"} err="failed to get container status \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": rpc error: code = NotFound desc = could not find container \"2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d\": container with ID starting with 2f6ee31f3660692b87eb07d87f4a0581bbd758ea3ec8e895d9a67ee5a203ca8d not found: ID does not exist" Jul 11 03:45:23.318213 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318139 2565 scope.go:117] "RemoveContainer" containerID="0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271" Jul 11 03:45:23.318382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318357 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271"} err="failed to get container status \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": rpc error: code = NotFound desc = could not find container \"0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271\": container with ID starting with 0d063433f07d99c81a40dfd73ecf4443ba33d01249bc0e2500a8e6c39229a271 not found: ID does not exist" Jul 11 03:45:23.318467 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318383 2565 scope.go:117] "RemoveContainer" containerID="3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135" Jul 11 03:45:23.318634 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318613 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135"} err="failed to get container status \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": rpc error: code = NotFound desc = could not find container \"3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135\": container with ID starting with 3dfeb4fb21dde83979f3f53f00e11cb28b975a6c48b6f77d8aaa6af895bc6135 not found: ID does not exist" Jul 11 03:45:23.318712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318639 2565 scope.go:117] "RemoveContainer" containerID="2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40" Jul 11 03:45:23.318712 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318700 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:23.318903 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318883 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40"} err="failed to get container status \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": rpc error: code = NotFound desc = could not find container \"2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40\": container with ID starting with 2341df1c5ad972e9a3267e7db7e31041283569eb77dbb4e73ff0688968a75e40 not found: ID does not exist" Jul 11 03:45:23.318984 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318905 2565 scope.go:117] "RemoveContainer" containerID="eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848" Jul 11 03:45:23.318984 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318975 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="init-config-reloader" Jul 11 03:45:23.319054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.318989 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="init-config-reloader" Jul 11 03:45:23.319054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319003 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-thanos" Jul 11 03:45:23.319054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319012 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-thanos" Jul 11 03:45:23.319054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319026 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-web" Jul 11 03:45:23.319054 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319034 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-web" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319060 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="prometheus" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319068 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="prometheus" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319077 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319085 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319096 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="config-reloader" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319105 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="config-reloader" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319115 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="thanos-sidecar" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319126 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="thanos-sidecar" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319164 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848"} err="failed to get container status \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": rpc error: code = NotFound desc = could not find container \"eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848\": container with ID starting with eef5c9431d7d61fe198d8e66112fce542436cab676ace8060281743ae7540848 not found: ID does not exist" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319178 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-thanos" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319187 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy-web" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319194 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="prometheus" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319199 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="kube-rbac-proxy" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319204 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="config-reloader" Jul 11 03:45:23.319214 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.319210 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" containerName="thanos-sidecar" Jul 11 03:45:23.324018 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.324003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.328218 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328197 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 11 03:45:23.328458 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 11 03:45:23.328534 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328457 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 11 03:45:23.328534 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328457 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6kc6388pii4va\"" Jul 11 03:45:23.328634 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328599 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 11 03:45:23.328634 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328571 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 11 03:45:23.328762 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328745 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 11 03:45:23.328879 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328862 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 11 03:45:23.328934 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.328884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 11 03:45:23.329038 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.329024 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c9mw5\"" Jul 11 03:45:23.329146 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.329119 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 11 03:45:23.329261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.329232 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 11 03:45:23.329503 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.329489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 11 03:45:23.330245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.330225 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 11 03:45:23.333561 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.333544 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 11 03:45:23.338357 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.338336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 11 03:45:23.361037 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.361009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:23.427395 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427395 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4fh\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-kube-api-access-xv4fh\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.427852 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.428097 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.428097 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.427899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.528970 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.528937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.528977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4fh\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-kube-api-access-xv4fh\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529428 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529428 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.529428 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.529209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.530658 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.530050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532124 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532091 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532236 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532236 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532236 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532402 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532636 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532897 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532814 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532897 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.532897 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.532871 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.533276 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.533246 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.533395 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.533370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.533457 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.533372 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.533618 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.533593 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.533995 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.533968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.534405 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.534353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.534727 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.534709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.535172 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.535153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.535893 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.535865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.535990 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.535964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.536046 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.535995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.536213 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.536199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-config\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.537810 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.537792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4fh\" (UniqueName: \"kubernetes.io/projected/5b86b755-2f4b-48f9-98dd-3d24c92c0a56-kube-api-access-xv4fh\") pod \"prometheus-k8s-0\" (UID: \"5b86b755-2f4b-48f9-98dd-3d24c92c0a56\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.633842 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.633796 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:45:23.769884 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:23.769867 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 03:45:23.772195 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:45:23.772083 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b86b755_2f4b_48f9_98dd_3d24c92c0a56.slice/crio-028fb26e8ac5ae9e68e17c9a2f86fe9e72aa899e638a6a4f1de58545dc1aae64 WatchSource:0}: Error finding container 028fb26e8ac5ae9e68e17c9a2f86fe9e72aa899e638a6a4f1de58545dc1aae64: Status 404 returned error can't find the container with id 028fb26e8ac5ae9e68e17c9a2f86fe9e72aa899e638a6a4f1de58545dc1aae64 Jul 11 03:45:24.263630 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:24.263594 2565 generic.go:356] "Generic (PLEG): container finished" podID="5b86b755-2f4b-48f9-98dd-3d24c92c0a56" containerID="63fdd3262cc3690b62c7de66ab00fbbfc479ec531fec586e4afb9e75ea21340e" exitCode=0 Jul 11 03:45:24.263834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:24.263645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerDied","Data":"63fdd3262cc3690b62c7de66ab00fbbfc479ec531fec586e4afb9e75ea21340e"} Jul 11 03:45:24.263834 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:24.263719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"028fb26e8ac5ae9e68e17c9a2f86fe9e72aa899e638a6a4f1de58545dc1aae64"} Jul 11 03:45:24.426549 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:24.426515 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f328fba0-fcb0-4671-94bc-b3a858340c80" path="/var/lib/kubelet/pods/f328fba0-fcb0-4671-94bc-b3a858340c80/volumes" Jul 11 03:45:25.270893 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.270851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"fe7b9405b688c202fb5ed7c65653fdb1191d3acb327a97ea19c0bf2befcb7ba3"} Jul 11 03:45:25.271245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.270935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"94ee970081da052a73f3bac8f5f778a6f1d4709707d2206c5665a10bd3f949be"} Jul 11 03:45:25.271245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.270975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"98873edc4571e836cc843c5498123411cd2bfe9cf2a950cc3c182796ed912313"} Jul 11 03:45:25.271245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.271014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"44cb2ab639795c8947e36237afe20ccdf5e6bf355855234a38940cf1ebfa79b4"} Jul 11 03:45:25.271245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.271052 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"41b4a52b48122208f37f02bfbfeebe2d984406d5a1de112ea1ec843abea419e8"} Jul 11 03:45:25.271245 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.271084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b86b755-2f4b-48f9-98dd-3d24c92c0a56","Type":"ContainerStarted","Data":"2424bb9f866e506c47144d711bd85060f9e461eb3dc7c2660f38705afbb1e291"} Jul 11 03:45:25.297853 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:25.297793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.277532536 podStartE2EDuration="2.297771432s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.238896ms" isStatefulPod=false podCreationTimestamp="2026-07-11 03:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:45:25.295948341 +0000 UTC m=+163.369564618" watchObservedRunningTime="2026-07-11 03:45:25.297771432 +0000 UTC m=+163.371387691" Jul 11 03:45:28.634334 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:45:28.634292 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:46:23.634581 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:46:23.634512 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:46:23.650766 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:46:23.650739 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:46:24.503031 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:46:24.502997 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 03:47:10.614597 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.614524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2"] Jul 11 03:47:10.618255 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.618236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.621100 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.621073 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 11 03:47:10.621222 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.621077 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 11 03:47:10.621222 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.621135 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 11 03:47:10.621222 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.621153 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2vrhl\"" Jul 11 03:47:10.621529 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.621514 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 11 03:47:10.637822 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.637795 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2"] Jul 11 03:47:10.772744 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.772670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.772906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.772781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.772906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.772804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69q7w\" (UniqueName: \"kubernetes.io/projected/553f0c20-fdda-4300-a1d1-50869171c0a2-kube-api-access-69q7w\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.772906 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.772833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.773014 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.772928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69q7w\" (UniqueName: \"kubernetes.io/projected/553f0c20-fdda-4300-a1d1-50869171c0a2-kube-api-access-69q7w\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873375 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873648 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.873957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.873926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-manifests\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.875960 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.875932 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/553f0c20-fdda-4300-a1d1-50869171c0a2-tmp\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.876063 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.876026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.876161 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.876145 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/553f0c20-fdda-4300-a1d1-50869171c0a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.881090 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.881058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69q7w\" (UniqueName: \"kubernetes.io/projected/553f0c20-fdda-4300-a1d1-50869171c0a2-kube-api-access-69q7w\") pod \"opendatahub-operator-controller-manager-7766d5cd97-scpc2\" (UID: \"553f0c20-fdda-4300-a1d1-50869171c0a2\") " pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:10.928142 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:10.928102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:11.063556 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:11.063480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2"] Jul 11 03:47:11.066230 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:47:11.066197 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553f0c20_fdda_4300_a1d1_50869171c0a2.slice/crio-66def2172268a4bfae114910a998338d470873cc564dd67dfff0f917aa1521dd WatchSource:0}: Error finding container 66def2172268a4bfae114910a998338d470873cc564dd67dfff0f917aa1521dd: Status 404 returned error can't find the container with id 66def2172268a4bfae114910a998338d470873cc564dd67dfff0f917aa1521dd Jul 11 03:47:11.657419 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:11.657367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" event={"ID":"553f0c20-fdda-4300-a1d1-50869171c0a2","Type":"ContainerStarted","Data":"66def2172268a4bfae114910a998338d470873cc564dd67dfff0f917aa1521dd"} Jul 11 03:47:14.671737 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:14.671701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" event={"ID":"553f0c20-fdda-4300-a1d1-50869171c0a2","Type":"ContainerStarted","Data":"68560501f027e829bb47d3a37d71f006f7e5d20b3ca6f69d11c3f6dfde4932ce"} Jul 11 03:47:15.677341 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:15.677303 2565 generic.go:356] "Generic (PLEG): container finished" podID="553f0c20-fdda-4300-a1d1-50869171c0a2" containerID="68560501f027e829bb47d3a37d71f006f7e5d20b3ca6f69d11c3f6dfde4932ce" exitCode=0 Jul 11 03:47:15.677830 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:15.677394 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" event={"ID":"553f0c20-fdda-4300-a1d1-50869171c0a2","Type":"ContainerDied","Data":"68560501f027e829bb47d3a37d71f006f7e5d20b3ca6f69d11c3f6dfde4932ce"} Jul 11 03:47:16.682610 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:16.682573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" event={"ID":"553f0c20-fdda-4300-a1d1-50869171c0a2","Type":"ContainerStarted","Data":"02c32734efda7077453d9fa1e6d3d8eba6610842b291f41bedfc6f4823286411"} Jul 11 03:47:16.683034 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:16.682729 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:16.703418 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:16.703360 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" podStartSLOduration=2.226648763 podStartE2EDuration="6.703341667s" totalImagesPullingTime="3.433449862s" totalInitContainerRuntime="1.043243042s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:16.701272004 +0000 UTC m=+274.774888272" watchObservedRunningTime="2026-07-11 03:47:16.703341667 +0000 UTC m=+274.776957927" Jul 11 03:47:27.688074 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:27.688040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7766d5cd97-scpc2" Jul 11 03:47:29.567435 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.567400 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z"] Jul 11 03:47:29.580295 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.580271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.583186 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.583158 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 11 03:47:29.583308 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.583228 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 11 03:47:29.583891 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.583864 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z"] Jul 11 03:47:29.583972 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.583919 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kvpwq\"" Jul 11 03:47:29.584231 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.584215 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 11 03:47:29.584270 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.584218 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 11 03:47:29.584306 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.584292 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 11 03:47:29.637008 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.636966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2q4l\" (UniqueName: \"kubernetes.io/projected/6615d324-fc39-4934-98e5-520f16407447-kube-api-access-z2q4l\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.637163 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.637038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.637163 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.637062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-metrics-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.637163 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.637084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6615d324-fc39-4934-98e5-520f16407447-manager-config\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.738246 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.738205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2q4l\" (UniqueName: \"kubernetes.io/projected/6615d324-fc39-4934-98e5-520f16407447-kube-api-access-z2q4l\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.738429 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.738302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.738429 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.738331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-metrics-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.738429 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.738363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6615d324-fc39-4934-98e5-520f16407447-manager-config\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.739637 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.739610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6615d324-fc39-4934-98e5-520f16407447-manager-config\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.741550 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.741525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.741664 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.741579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6615d324-fc39-4934-98e5-520f16407447-metrics-cert\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.749415 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.749383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2q4l\" (UniqueName: \"kubernetes.io/projected/6615d324-fc39-4934-98e5-520f16407447-kube-api-access-z2q4l\") pod \"lws-controller-manager-6d946858c4-mnx4z\" (UID: \"6615d324-fc39-4934-98e5-520f16407447\") " pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:29.890223 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:29.890182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:30.029950 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:30.029917 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z"] Jul 11 03:47:30.032957 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:47:30.032930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6615d324_fc39_4934_98e5_520f16407447.slice/crio-e7ac3994e07d5b21d49a0e44faace3093d9a73fdb0c14d6b36c8963c8a9a7f3c WatchSource:0}: Error finding container e7ac3994e07d5b21d49a0e44faace3093d9a73fdb0c14d6b36c8963c8a9a7f3c: Status 404 returned error can't find the container with id e7ac3994e07d5b21d49a0e44faace3093d9a73fdb0c14d6b36c8963c8a9a7f3c Jul 11 03:47:30.742369 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:30.742316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" event={"ID":"6615d324-fc39-4934-98e5-520f16407447","Type":"ContainerStarted","Data":"e7ac3994e07d5b21d49a0e44faace3093d9a73fdb0c14d6b36c8963c8a9a7f3c"} Jul 11 03:47:31.101389 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.101309 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4rqqt"] Jul 11 03:47:31.104602 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.104579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.107920 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.107877 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fpnnt\"" Jul 11 03:47:31.107920 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.107890 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 11 03:47:31.115602 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.115560 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4rqqt"] Jul 11 03:47:31.150210 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.150174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8zt\" (UniqueName: \"kubernetes.io/projected/122035e6-6c01-4b22-8d93-68b677a7c69d-kube-api-access-lx8zt\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.150385 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.150273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122035e6-6c01-4b22-8d93-68b677a7c69d-tls-certs\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.250994 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.250950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8zt\" (UniqueName: \"kubernetes.io/projected/122035e6-6c01-4b22-8d93-68b677a7c69d-kube-api-access-lx8zt\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.251242 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.251014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122035e6-6c01-4b22-8d93-68b677a7c69d-tls-certs\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.254123 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.254096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122035e6-6c01-4b22-8d93-68b677a7c69d-tls-certs\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.259084 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.259055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8zt\" (UniqueName: \"kubernetes.io/projected/122035e6-6c01-4b22-8d93-68b677a7c69d-kube-api-access-lx8zt\") pod \"model-serving-api-54d89bd5fc-4rqqt\" (UID: \"122035e6-6c01-4b22-8d93-68b677a7c69d\") " pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:31.418644 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:31.418605 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:32.378606 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:32.378537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4rqqt"] Jul 11 03:47:32.380882 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:47:32.380852 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122035e6_6c01_4b22_8d93_68b677a7c69d.slice/crio-798fd81ab2245de603898cca66ecaf526293b10ddd8f5a055be797e0ae614674 WatchSource:0}: Error finding container 798fd81ab2245de603898cca66ecaf526293b10ddd8f5a055be797e0ae614674: Status 404 returned error can't find the container with id 798fd81ab2245de603898cca66ecaf526293b10ddd8f5a055be797e0ae614674 Jul 11 03:47:32.752175 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:32.752136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" event={"ID":"6615d324-fc39-4934-98e5-520f16407447","Type":"ContainerStarted","Data":"3970aa5fe7a699c6e732c3d107c182cd72842a41a3a8dbaadb54cae19fef887b"} Jul 11 03:47:32.752346 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:32.752244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:32.753238 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:32.753217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" event={"ID":"122035e6-6c01-4b22-8d93-68b677a7c69d","Type":"ContainerStarted","Data":"798fd81ab2245de603898cca66ecaf526293b10ddd8f5a055be797e0ae614674"} Jul 11 03:47:32.768290 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:32.768239 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" podStartSLOduration=1.4907592570000001 podStartE2EDuration="3.768224293s" totalImagesPullingTime="2.277465036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:32.766888798 +0000 UTC m=+290.840505056" watchObservedRunningTime="2026-07-11 03:47:32.768224293 +0000 UTC m=+290.841840551" Jul 11 03:47:34.763596 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:34.763549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" event={"ID":"122035e6-6c01-4b22-8d93-68b677a7c69d","Type":"ContainerStarted","Data":"dbe40d98eececfaf9aebbe8c4032359f3782e3dbd6b1672031cbd19e0df6adba"} Jul 11 03:47:34.764098 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:34.763672 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:47:34.779847 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:34.779797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" podStartSLOduration=2.234365279 podStartE2EDuration="3.779778073s" totalImagesPullingTime="1.545412794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:34.778817022 +0000 UTC m=+292.852433282" watchObservedRunningTime="2026-07-11 03:47:34.779778073 +0000 UTC m=+292.853394332" Jul 11 03:47:36.900018 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.899983 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk"] Jul 11 03:47:36.903254 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.903237 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.905660 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.905635 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fnmdk\"" Jul 11 03:47:36.905815 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.905638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 11 03:47:36.917039 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.917012 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk"] Jul 11 03:47:36.995148 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995148 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995371 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsht\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-kube-api-access-jzsht\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995371 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995371 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995480 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995480 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995602 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:36.995602 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:36.995524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0201f665-6c8b-4e08-a54d-83555b2a5f50-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096139 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0201f665-6c8b-4e08-a54d-83555b2a5f50-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsht\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-kube-api-access-jzsht\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096709 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096709 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096807 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.096971 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.096945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.097095 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.097070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.097441 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.097373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0201f665-6c8b-4e08-a54d-83555b2a5f50-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.099034 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.099016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.099196 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.099179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.104881 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.104854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.104972 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.104948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsht\" (UniqueName: \"kubernetes.io/projected/0201f665-6c8b-4e08-a54d-83555b2a5f50-kube-api-access-jzsht\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cqbnk\" (UID: \"0201f665-6c8b-4e08-a54d-83555b2a5f50\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.214905 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.214811 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:37.349356 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.349329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk"] Jul 11 03:47:37.351405 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:47:37.351370 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201f665_6c8b_4e08_a54d_83555b2a5f50.slice/crio-161312e36b94230f5464f61dda44e68c5cc3bb62a95c5f8658327e32e183ddbe WatchSource:0}: Error finding container 161312e36b94230f5464f61dda44e68c5cc3bb62a95c5f8658327e32e183ddbe: Status 404 returned error can't find the container with id 161312e36b94230f5464f61dda44e68c5cc3bb62a95c5f8658327e32e183ddbe Jul 11 03:47:37.777505 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.777470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" event={"ID":"0201f665-6c8b-4e08-a54d-83555b2a5f50","Type":"ContainerStarted","Data":"161312e36b94230f5464f61dda44e68c5cc3bb62a95c5f8658327e32e183ddbe"} Jul 11 03:47:37.865762 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.865721 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4"] Jul 11 03:47:37.870259 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.870233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.873329 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.873298 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 11 03:47:37.873485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.873299 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 11 03:47:37.873485 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.873336 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 11 03:47:37.873742 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.873727 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bch8v\"" Jul 11 03:47:37.884820 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.884791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4"] Jul 11 03:47:37.903359 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c44c2-ff35-4725-9d99-6447d192da92-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903359 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903809 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4zr\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-kube-api-access-dc4zr\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903809 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903809 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903809 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:37.903809 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:37.903640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.004721 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.004662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c44c2-ff35-4725-9d99-6447d192da92-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.004901 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.004740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.004901 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.004801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4zr\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-kube-api-access-dc4zr\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.004901 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.004844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.005055 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.004959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.005055 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.005023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.005150 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.005058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.006612 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.006392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.007527 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.007501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70c44c2-ff35-4725-9d99-6447d192da92-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.008223 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.008199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.008514 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.008420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.008643 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.008570 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70c44c2-ff35-4725-9d99-6447d192da92-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.013431 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.013400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.013807 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.013782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4zr\" (UniqueName: \"kubernetes.io/projected/e70c44c2-ff35-4725-9d99-6447d192da92-kube-api-access-dc4zr\") pod \"istiod-openshift-gateway-94bb8fbfd-csxj4\" (UID: \"e70c44c2-ff35-4725-9d99-6447d192da92\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.180480 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.180414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:38.480139 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.480062 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4"] Jul 11 03:47:38.483006 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:47:38.482974 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c44c2_ff35_4725_9d99_6447d192da92.slice/crio-48fe01c423ffb8215deff1945409c7f4fd78e4eca973fca09d210e13bcbc230a WatchSource:0}: Error finding container 48fe01c423ffb8215deff1945409c7f4fd78e4eca973fca09d210e13bcbc230a: Status 404 returned error can't find the container with id 48fe01c423ffb8215deff1945409c7f4fd78e4eca973fca09d210e13bcbc230a Jul 11 03:47:38.784411 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:38.784316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" event={"ID":"e70c44c2-ff35-4725-9d99-6447d192da92","Type":"ContainerStarted","Data":"48fe01c423ffb8215deff1945409c7f4fd78e4eca973fca09d210e13bcbc230a"} Jul 11 03:47:41.074611 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.074554 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 03:47:41.074898 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.074655 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 03:47:41.074898 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.074718 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 03:47:41.075454 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.075426 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 03:47:41.075516 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.075483 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 03:47:41.802042 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.801998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" event={"ID":"e70c44c2-ff35-4725-9d99-6447d192da92","Type":"ContainerStarted","Data":"f2184fffb77051b25e4ea478e9fa1c06ee1b953dabebdb588911271e186652db"} Jul 11 03:47:41.802347 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.802126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:41.803553 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.803525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" event={"ID":"0201f665-6c8b-4e08-a54d-83555b2a5f50","Type":"ContainerStarted","Data":"6c2a7d3d2fc4c02b568ee0636206a13d061105e688bbd6c4038a3baf72047d4e"} Jul 11 03:47:41.848042 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.847988 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" podStartSLOduration=2.258374187 podStartE2EDuration="4.847972214s" totalImagesPullingTime="2.589598027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:41.847807439 +0000 UTC m=+299.921423697" watchObservedRunningTime="2026-07-11 03:47:41.847972214 +0000 UTC m=+299.921588473" Jul 11 03:47:41.887525 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:41.887462 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" podStartSLOduration=2.166833217 podStartE2EDuration="5.887442163s" totalImagesPullingTime="3.720608946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:47:41.884959337 +0000 UTC m=+299.958575596" watchObservedRunningTime="2026-07-11 03:47:41.887442163 +0000 UTC m=+299.961058422" Jul 11 03:47:42.215755 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.215717 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:42.220173 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.220152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:42.372946 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.372921 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:47:42.373353 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.373329 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:47:42.377249 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.377227 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 11 03:47:42.808545 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.808514 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:42.810761 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.809235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cqbnk" Jul 11 03:47:42.810761 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:42.810265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-csxj4" Jul 11 03:47:43.760613 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:43.760585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d946858c4-mnx4z" Jul 11 03:47:45.772112 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:47:45.772081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4rqqt" Jul 11 03:50:26.186050 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.186015 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ptw86"] Jul 11 03:50:26.189690 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.189658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.192040 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.192018 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tsw7r\"" Jul 11 03:50:26.192174 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.192025 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 11 03:50:26.192174 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.192086 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 11 03:50:26.192567 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.192551 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 11 03:50:26.198334 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.198315 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ptw86"] Jul 11 03:50:26.327315 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.327268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-data\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.327522 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.327441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv675\" (UniqueName: \"kubernetes.io/projected/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-kube-api-access-xv675\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.427914 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.427884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv675\" (UniqueName: \"kubernetes.io/projected/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-kube-api-access-xv675\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.428104 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.427927 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-data\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.428261 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.428242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-data\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.435943 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.435917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv675\" (UniqueName: \"kubernetes.io/projected/dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309-kube-api-access-xv675\") pod \"postgres-69fb4594cf-ptw86\" (UID: \"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.501382 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.501300 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:26.629454 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.629430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ptw86"] Jul 11 03:50:26.632066 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:50:26.632041 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc9f3a2_b1ea_4cf7_96f5_f911ba8a6309.slice/crio-1dae292ec367c8e3b1e561dac19b777ff58272c10742b794d02d783023e401c8 WatchSource:0}: Error finding container 1dae292ec367c8e3b1e561dac19b777ff58272c10742b794d02d783023e401c8: Status 404 returned error can't find the container with id 1dae292ec367c8e3b1e561dac19b777ff58272c10742b794d02d783023e401c8 Jul 11 03:50:26.633364 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:26.633348 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 11 03:50:27.485087 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:27.485043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" event={"ID":"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309","Type":"ContainerStarted","Data":"1dae292ec367c8e3b1e561dac19b777ff58272c10742b794d02d783023e401c8"} Jul 11 03:50:31.504315 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:31.504278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" event={"ID":"dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309","Type":"ContainerStarted","Data":"4d9edd641b274d41ddb55eebe3e74f7a84d2acafb4306a7c67335d64ff6b13d2"} Jul 11 03:50:31.504707 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:31.504401 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:50:31.519386 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:31.519341 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" podStartSLOduration=0.802509632 podStartE2EDuration="5.51932584s" totalImagesPullingTime="4.716816208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:50:31.517934534 +0000 UTC m=+469.591550794" watchObservedRunningTime="2026-07-11 03:50:31.51932584 +0000 UTC m=+469.592942102" Jul 11 03:50:37.538486 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:50:37.538455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ptw86" Jul 11 03:51:20.164572 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.164533 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-97mdg"] Jul 11 03:51:20.168275 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.168257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.170406 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.170384 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 11 03:51:20.177264 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.177227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-97mdg"] Jul 11 03:51:20.316443 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.316398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crccj\" (UniqueName: \"kubernetes.io/projected/4195a265-12ba-472d-baca-fcdd164c577a-kube-api-access-crccj\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.316617 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.316456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4195a265-12ba-472d-baca-fcdd164c577a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.417933 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.417835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crccj\" (UniqueName: \"kubernetes.io/projected/4195a265-12ba-472d-baca-fcdd164c577a-kube-api-access-crccj\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.417933 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.417884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4195a265-12ba-472d-baca-fcdd164c577a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.418606 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.418586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4195a265-12ba-472d-baca-fcdd164c577a-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.426300 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.426271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crccj\" (UniqueName: \"kubernetes.io/projected/4195a265-12ba-472d-baca-fcdd164c577a-kube-api-access-crccj\") pod \"payload-pre-processing-74584bb9f8-97mdg\" (UID: \"4195a265-12ba-472d-baca-fcdd164c577a\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.485254 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.485218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:20.620420 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.620386 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-97mdg"] Jul 11 03:51:20.623534 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:51:20.623500 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4195a265_12ba_472d_baca_fcdd164c577a.slice/crio-f9f337ed25c7e22a03c1ef35def66bf6a4a430f18bbf02d84ad7af46db088173 WatchSource:0}: Error finding container f9f337ed25c7e22a03c1ef35def66bf6a4a430f18bbf02d84ad7af46db088173: Status 404 returned error can't find the container with id f9f337ed25c7e22a03c1ef35def66bf6a4a430f18bbf02d84ad7af46db088173 Jul 11 03:51:20.704387 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:20.704305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" event={"ID":"4195a265-12ba-472d-baca-fcdd164c577a","Type":"ContainerStarted","Data":"f9f337ed25c7e22a03c1ef35def66bf6a4a430f18bbf02d84ad7af46db088173"} Jul 11 03:51:22.714804 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:22.714769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" event={"ID":"4195a265-12ba-472d-baca-fcdd164c577a","Type":"ContainerStarted","Data":"4a0468aed1201d523ffd6de363943c284c2acc019a7b0ea24fbc872e6be0ce8b"} Jul 11 03:51:22.715246 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:22.714839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:51:22.730153 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:22.730102 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" podStartSLOduration=1.445003409 podStartE2EDuration="2.730079406s" totalImagesPullingTime="1.285075997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:51:22.727924699 +0000 UTC m=+520.801540958" watchObservedRunningTime="2026-07-11 03:51:22.730079406 +0000 UTC m=+520.803695665" Jul 11 03:51:33.720619 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:33.720516 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" podUID="4195a265-12ba-472d-baca-fcdd164c577a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.27:9004: connect: connection refused" Jul 11 03:51:43.720149 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:51:43.720118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-97mdg" Jul 11 03:52:13.028592 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.028554 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j"] Jul 11 03:52:13.031878 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.031860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.035957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.035930 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 11 03:52:13.035957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.035947 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 11 03:52:13.035957 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.035933 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 11 03:52:13.036210 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.035936 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lvfkx\"" Jul 11 03:52:13.048034 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.048009 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j"] Jul 11 03:52:13.092011 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.091976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f0898-2825-47b2-be29-cfbfc0372a82-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.092011 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.092014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.092248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.092043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grccj\" (UniqueName: \"kubernetes.io/projected/b30f0898-2825-47b2-be29-cfbfc0372a82-kube-api-access-grccj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.092248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.092062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.092248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.092172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.092248 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.092211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.192921 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.192881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f0898-2825-47b2-be29-cfbfc0372a82-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.192921 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.192918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.192954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grccj\" (UniqueName: \"kubernetes.io/projected/b30f0898-2825-47b2-be29-cfbfc0372a82-kube-api-access-grccj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.192973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.193020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193145 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.193036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193406 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.193386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193458 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.193415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.193512 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.193467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.195421 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.195404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30f0898-2825-47b2-be29-cfbfc0372a82-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.195544 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.195527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30f0898-2825-47b2-be29-cfbfc0372a82-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.200603 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.200580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grccj\" (UniqueName: \"kubernetes.io/projected/b30f0898-2825-47b2-be29-cfbfc0372a82-kube-api-access-grccj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sxr7j\" (UID: \"b30f0898-2825-47b2-be29-cfbfc0372a82\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.342274 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.342181 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:13.479717 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.479670 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j"] Jul 11 03:52:13.482421 ip-10-0-138-240 kubenswrapper[2565]: W0711 03:52:13.482393 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30f0898_2825_47b2_be29_cfbfc0372a82.slice/crio-63924a0a555a1d51edc32ddc73ab07f42d2cc731b7fa59eb16dbe33774246bcf WatchSource:0}: Error finding container 63924a0a555a1d51edc32ddc73ab07f42d2cc731b7fa59eb16dbe33774246bcf: Status 404 returned error can't find the container with id 63924a0a555a1d51edc32ddc73ab07f42d2cc731b7fa59eb16dbe33774246bcf Jul 11 03:52:13.942084 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:13.942021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" event={"ID":"b30f0898-2825-47b2-be29-cfbfc0372a82","Type":"ContainerStarted","Data":"63924a0a555a1d51edc32ddc73ab07f42d2cc731b7fa59eb16dbe33774246bcf"} Jul 11 03:52:14.949134 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:14.949101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" event={"ID":"b30f0898-2825-47b2-be29-cfbfc0372a82","Type":"ContainerStarted","Data":"d2d804627fb8bf36a240206420f687e5b4f26981ed991ecee971fb1e5ce023df"} Jul 11 03:52:14.968474 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:14.968406 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" podStartSLOduration=1.965809938 podStartE2EDuration="2.968386199s" totalImagesPullingTime="1.002576261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 03:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 03:52:14.968049517 +0000 UTC m=+573.041665775" watchObservedRunningTime="2026-07-11 03:52:14.968386199 +0000 UTC m=+573.042002460" Jul 11 03:52:23.342500 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:23.342457 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:23.343025 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:23.342535 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:23.354853 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:23.354818 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:23.999273 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:23.999246 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sxr7j" Jul 11 03:52:42.398322 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:42.398290 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:52:42.399121 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:52:42.399103 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:57:42.422858 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:57:42.422772 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 03:57:42.426192 ip-10-0-138-240 kubenswrapper[2565]: I0711 03:57:42.426168 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:02:42.448220 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:02:42.448138 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:02:42.451940 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:02:42.451919 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:07:16.952894 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:16.952858 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:16.956614 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:16.956594 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:16.958997 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:16.958977 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-af4c0929-openshift-default-dockercfg-kwpv7\"" Jul 11 04:07:16.979669 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:16.979632 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:17.026744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.026909 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.026909 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.026909 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.027013 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.027013 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.027013 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.026993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzbk\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.027105 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.027018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.027105 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.027044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127723 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzbk\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.127901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128257 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.127920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128257 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.128200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128257 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.128208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128496 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.128473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128572 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.128550 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.128672 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.128645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.130126 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.130105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.130690 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.130651 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.136235 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.136214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzbk\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.136403 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.136381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token\") pod \"e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.267333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.267242 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:17.410356 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.410333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:17.412627 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:07:17.412599 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ff0d3b_cc37_4a1a_b07a_387106ac685b.slice/crio-0da6bb7889acb01beb373f7f241fea42f7422c37c9f103a48c729c1c4d8596a4 WatchSource:0}: Error finding container 0da6bb7889acb01beb373f7f241fea42f7422c37c9f103a48c729c1c4d8596a4: Status 404 returned error can't find the container with id 0da6bb7889acb01beb373f7f241fea42f7422c37c9f103a48c729c1c4d8596a4 Jul 11 04:07:17.415091 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.415048 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:17.415200 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.415135 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:17.415200 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.415176 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:17.888726 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.888657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerStarted","Data":"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321"} Jul 11 04:07:17.888903 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.888743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerStarted","Data":"0da6bb7889acb01beb373f7f241fea42f7422c37c9f103a48c729c1c4d8596a4"} Jul 11 04:07:17.907745 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:17.907688 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podStartSLOduration=1.907655658 podStartE2EDuration="1.907655658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:07:17.906061796 +0000 UTC m=+1475.979678058" watchObservedRunningTime="2026-07-11 04:07:17.907655658 +0000 UTC m=+1475.981271916" Jul 11 04:07:18.267764 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:18.267663 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:18.274357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:18.274327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:18.893474 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:18.893436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:18.894395 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:18.894375 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:23.100606 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.100569 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx"] Jul 11 04:07:23.104223 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.104200 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:23.106306 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.106287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:07:23.114700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.111612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx"] Jul 11 04:07:23.184110 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.184070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx\" (UID: \"c0959a69-c706-4f40-9ccc-b1e18302f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:23.285086 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.285040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx\" (UID: \"c0959a69-c706-4f40-9ccc-b1e18302f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:23.285652 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.285629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx\" (UID: \"c0959a69-c706-4f40-9ccc-b1e18302f928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:23.428081 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.428047 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:23.601022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.600991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx"] Jul 11 04:07:23.604842 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:07:23.604816 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0959a69_c706_4f40_9ccc_b1e18302f928.slice/crio-30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a WatchSource:0}: Error finding container 30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a: Status 404 returned error can't find the container with id 30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a Jul 11 04:07:23.607108 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.607093 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 11 04:07:23.917509 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:23.917469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" event={"ID":"c0959a69-c706-4f40-9ccc-b1e18302f928","Type":"ContainerStarted","Data":"30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a"} Jul 11 04:07:26.935404 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:26.934559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" event={"ID":"c0959a69-c706-4f40-9ccc-b1e18302f928","Type":"ContainerStarted","Data":"729122c16ae2e7dfecf82abd576bba5ffed85491dad05b8b4212fde74b3da4d2"} Jul 11 04:07:27.942181 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:27.942141 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0959a69-c706-4f40-9ccc-b1e18302f928" containerID="729122c16ae2e7dfecf82abd576bba5ffed85491dad05b8b4212fde74b3da4d2" exitCode=0 Jul 11 04:07:27.942714 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:27.942217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" event={"ID":"c0959a69-c706-4f40-9ccc-b1e18302f928","Type":"ContainerDied","Data":"729122c16ae2e7dfecf82abd576bba5ffed85491dad05b8b4212fde74b3da4d2"} Jul 11 04:07:29.088520 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.088496 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:29.148844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.148808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca\") pod \"c0959a69-c706-4f40-9ccc-b1e18302f928\" (UID: \"c0959a69-c706-4f40-9ccc-b1e18302f928\") " Jul 11 04:07:29.149200 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.149172 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0959a69-c706-4f40-9ccc-b1e18302f928" (UID: "c0959a69-c706-4f40-9ccc-b1e18302f928"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:07:29.249755 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.249636 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0959a69-c706-4f40-9ccc-b1e18302f928-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:29.953884 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.953848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" event={"ID":"c0959a69-c706-4f40-9ccc-b1e18302f928","Type":"ContainerDied","Data":"30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a"} Jul 11 04:07:29.954082 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.953908 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30bf1cfab41c94f765436bb55df911f2c21ab8338f79dc1068e54f05dd1f755a" Jul 11 04:07:29.954082 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:29.953862 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx" Jul 11 04:07:30.959475 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.959446 2565 generic.go:356] "Generic (PLEG): container finished" podID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerID="7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321" exitCode=0 Jul 11 04:07:30.959931 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.959522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerDied","Data":"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321"} Jul 11 04:07:30.959973 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.959951 2565 scope.go:117] "RemoveContainer" containerID="7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321" Jul 11 04:07:30.960971 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.960945 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:30.961042 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.961009 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:30.961081 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:30.961043 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:31.966304 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:31.966270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerStarted","Data":"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8"} Jul 11 04:07:32.268512 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:32.268409 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:32.269608 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:32.269570 2565 patch_prober.go:28] interesting pod/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 11 04:07:32.269750 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:32.269630 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 11 04:07:33.268025 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:33.267984 2565 patch_prober.go:28] interesting pod/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 11 04:07:33.268418 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:33.268041 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 11 04:07:33.894124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:33.894082 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:34.268419 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:34.268317 2565 patch_prober.go:28] interesting pod/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 11 04:07:34.268419 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:34.268379 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 11 04:07:35.268720 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:35.268651 2565 patch_prober.go:28] interesting pod/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 11 04:07:35.269205 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:35.268760 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 11 04:07:36.272362 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:36.272329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:36.273478 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:36.273461 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:42.475151 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:42.475117 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:07:42.478992 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:42.478971 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:07:48.347095 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.347029 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:48.347662 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.347230 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" containerID="cri-o://82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8" gracePeriod=30 Jul 11 04:07:48.916774 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.916740 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:07:48.917364 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.917337 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0959a69-c706-4f40-9ccc-b1e18302f928" containerName="revoke-keys" Jul 11 04:07:48.917364 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.917364 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0959a69-c706-4f40-9ccc-b1e18302f928" containerName="revoke-keys" Jul 11 04:07:48.917546 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.917465 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0959a69-c706-4f40-9ccc-b1e18302f928" containerName="revoke-keys" Jul 11 04:07:48.920490 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.920469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:48.922874 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.922851 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4b08374c-gw-openshift-default-dockercfg-x24mm\"" Jul 11 04:07:48.933787 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:48.933761 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:07:49.044491 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044748 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044748 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044748 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044920 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044920 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044920 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.044920 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.045089 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.044943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfgs\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfgs\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.146770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.146745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.147123 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.147037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.147123 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.147087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.147218 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.147153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.147218 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.147164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.147397 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.147379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.149108 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.149086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.149322 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.149304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.154267 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.154244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.154351 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.154273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfgs\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs\") pod \"e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.231710 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.231562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:49.369938 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.369866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:07:49.372589 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:07:49.372562 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b835bc0_6589_41b6_83b3_26626855ab44.slice/crio-548cf0c05bf14fffb5618f65d8c354359dddf66b10fc16bf221ecdc0566ca4f2 WatchSource:0}: Error finding container 548cf0c05bf14fffb5618f65d8c354359dddf66b10fc16bf221ecdc0566ca4f2: Status 404 returned error can't find the container with id 548cf0c05bf14fffb5618f65d8c354359dddf66b10fc16bf221ecdc0566ca4f2 Jul 11 04:07:49.374865 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.374828 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:49.374955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.374900 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:49.374955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:49.374934 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:07:50.049979 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:50.049941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerStarted","Data":"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca"} Jul 11 04:07:50.050169 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:50.049997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerStarted","Data":"548cf0c05bf14fffb5618f65d8c354359dddf66b10fc16bf221ecdc0566ca4f2"} Jul 11 04:07:50.067646 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:50.067578 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podStartSLOduration=2.067559408 podStartE2EDuration="2.067559408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:07:50.066767272 +0000 UTC m=+1508.140383548" watchObservedRunningTime="2026-07-11 04:07:50.067559408 +0000 UTC m=+1508.141175667" Jul 11 04:07:50.232144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:50.232104 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:50.237067 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:50.237042 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:51.055283 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:51.055254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:51.056198 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:51.056180 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:07:51.275621 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:51.275576 2565 patch_prober.go:28] interesting pod/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:07:51.275802 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:51.275650 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:07:53.605394 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.605370 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:53.693003 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.692977 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693122 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693122 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693122 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693064 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693254 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693254 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693198 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgzbk\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693254 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693229 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693409 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693254 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693409 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693287 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:07:53.693409 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy\") pod \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\" (UID: \"64ff0d3b-cc37-4a1a-b07a-387106ac685b\") " Jul 11 04:07:53.693554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693464 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:07:53.693823 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693796 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.693932 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693825 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.693999 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.693882 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:07:53.694134 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.694103 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:07:53.694735 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.694709 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data" (OuterVolumeSpecName: "istio-data") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:07:53.695705 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.695649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:07:53.695818 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.695746 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk" (OuterVolumeSpecName: "kube-api-access-sgzbk") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "kube-api-access-sgzbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:07:53.695818 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.695757 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token" (OuterVolumeSpecName: "istio-token") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:07:53.695932 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.695819 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64ff0d3b-cc37-4a1a-b07a-387106ac685b" (UID: "64ff0d3b-cc37-4a1a-b07a-387106ac685b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:07:53.794779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794748 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzbk\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-kube-api-access-sgzbk\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794776 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794788 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794998 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794797 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794998 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794805 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794998 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794812 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:53.794998 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:53.794820 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64ff0d3b-cc37-4a1a-b07a-387106ac685b-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:07:54.070662 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.070576 2565 generic.go:356] "Generic (PLEG): container finished" podID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerID="82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8" exitCode=0 Jul 11 04:07:54.070662 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.070653 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" Jul 11 04:07:54.070843 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.070663 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerDied","Data":"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8"} Jul 11 04:07:54.070843 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.070786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9" event={"ID":"64ff0d3b-cc37-4a1a-b07a-387106ac685b","Type":"ContainerDied","Data":"0da6bb7889acb01beb373f7f241fea42f7422c37c9f103a48c729c1c4d8596a4"} Jul 11 04:07:54.070843 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.070819 2565 scope.go:117] "RemoveContainer" containerID="82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8" Jul 11 04:07:54.080871 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.080852 2565 scope.go:117] "RemoveContainer" containerID="7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321" Jul 11 04:07:54.088883 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.088859 2565 scope.go:117] "RemoveContainer" containerID="82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8" Jul 11 04:07:54.089314 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:07:54.089249 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8\": container with ID starting with 82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8 not found: ID does not exist" containerID="82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8" Jul 11 04:07:54.089314 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.089295 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8"} err="failed to get container status \"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8\": rpc error: code = NotFound desc = could not find container \"82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8\": container with ID starting with 82c21d78da58cd221d51ab79b737e802eccd5b9fc7036cd18c15805a089645f8 not found: ID does not exist" Jul 11 04:07:54.089314 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.089317 2565 scope.go:117] "RemoveContainer" containerID="7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321" Jul 11 04:07:54.089644 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:07:54.089582 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321\": container with ID starting with 7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321 not found: ID does not exist" containerID="7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321" Jul 11 04:07:54.089644 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.089611 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321"} err="failed to get container status \"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321\": rpc error: code = NotFound desc = could not find container \"7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321\": container with ID starting with 7769c2046bf491bc11620f980082bafb9c119b796eb044dfb1878cb54a210321 not found: ID does not exist" Jul 11 04:07:54.091458 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.091436 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:54.094641 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.094618 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-af4c0929-openshift-default-7c8cd6548c-qvsf9"] Jul 11 04:07:54.426027 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.425987 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" path="/var/lib/kubelet/pods/64ff0d3b-cc37-4a1a-b07a-387106ac685b/volumes" Jul 11 04:07:54.723047 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.722953 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:07:54.723500 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723486 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.723566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723508 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.723566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723525 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.723566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723534 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.723728 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723617 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.723728 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.723630 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ff0d3b-cc37-4a1a-b07a-387106ac685b" containerName="istio-proxy" Jul 11 04:07:54.728339 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.728306 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:07:54.730610 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.730581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:07:54.732688 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.732649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:07:54.904103 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:54.904062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn\" (UID: \"a2c744b8-97c0-46cc-a378-a888f131e23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:07:55.004667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:55.004574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn\" (UID: \"a2c744b8-97c0-46cc-a378-a888f131e23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:07:55.005213 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:55.005192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn\" (UID: \"a2c744b8-97c0-46cc-a378-a888f131e23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:07:55.040727 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:55.040659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:07:55.174144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:55.174116 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:07:55.175792 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:07:55.175763 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c744b8_97c0_46cc_a378_a888f131e23f.slice/crio-5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3 WatchSource:0}: Error finding container 5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3: Status 404 returned error can't find the container with id 5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3 Jul 11 04:07:56.082382 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:56.082345 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerID="fca7aff817c80f7e9c125357b18a57e35e5019bc5d13543370c7140ef41c46ab" exitCode=7 Jul 11 04:07:56.082771 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:56.082434 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerDied","Data":"fca7aff817c80f7e9c125357b18a57e35e5019bc5d13543370c7140ef41c46ab"} Jul 11 04:07:56.082771 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:56.082487 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerStarted","Data":"5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3"} Jul 11 04:07:56.082771 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:56.082562 2565 scope.go:117] "RemoveContainer" containerID="fca7aff817c80f7e9c125357b18a57e35e5019bc5d13543370c7140ef41c46ab" Jul 11 04:07:57.089021 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:57.088943 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerID="1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045" exitCode=7 Jul 11 04:07:57.089021 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:57.089002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerDied","Data":"1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045"} Jul 11 04:07:57.089481 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:57.089050 2565 scope.go:117] "RemoveContainer" containerID="fca7aff817c80f7e9c125357b18a57e35e5019bc5d13543370c7140ef41c46ab" Jul 11 04:07:57.089481 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:57.089250 2565 scope.go:117] "RemoveContainer" containerID="1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045" Jul 11 04:07:57.089481 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:07:57.089470 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn_odh-ai-gateway-infra(a2c744b8-97c0-46cc-a378-a888f131e23f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" Jul 11 04:07:58.095425 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:07:58.095400 2565 scope.go:117] "RemoveContainer" containerID="1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045" Jul 11 04:07:58.095815 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:07:58.095620 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn_odh-ai-gateway-infra(a2c744b8-97c0-46cc-a378-a888f131e23f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" Jul 11 04:08:05.127981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.127886 2565 generic.go:356] "Generic (PLEG): container finished" podID="1b835bc0-6589-41b6-83b3-26626855ab44" containerID="924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca" exitCode=0 Jul 11 04:08:05.127981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.127955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerDied","Data":"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca"} Jul 11 04:08:05.128416 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.128339 2565 scope.go:117] "RemoveContainer" containerID="924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca" Jul 11 04:08:05.129276 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.129245 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:08:05.129378 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.129319 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:08:05.129378 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:05.129361 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:08:06.055296 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:06.055249 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:08:06.134383 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:06.134337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerStarted","Data":"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51"} Jul 11 04:08:06.232313 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:06.232279 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:08:06.233387 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:06.233358 2565 patch_prober.go:28] interesting pod/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 11 04:08:06.233508 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:06.233445 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 11 04:08:07.232947 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:07.232908 2565 patch_prober.go:28] interesting pod/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 11 04:08:07.233335 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:07.232962 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 11 04:08:08.232269 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:08.232236 2565 patch_prober.go:28] interesting pod/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 11 04:08:08.232452 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:08.232293 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 11 04:08:09.232153 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:09.232108 2565 patch_prober.go:28] interesting pod/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 11 04:08:09.232629 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:09.232187 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 11 04:08:09.421207 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:09.421172 2565 scope.go:117] "RemoveContainer" containerID="1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045" Jul 11 04:08:10.160812 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:10.160782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerStarted","Data":"6b5e16f179411429ca145175702749280acc4348b0ee2d4b85ec7e8ec047149f"} Jul 11 04:08:10.177552 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:10.177495 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" podStartSLOduration=16.17747818 podStartE2EDuration="16.17747818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:08:10.175741125 +0000 UTC m=+1528.249357455" watchObservedRunningTime="2026-07-11 04:08:10.17747818 +0000 UTC m=+1528.251094439" Jul 11 04:08:10.236771 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:10.236741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:08:10.237186 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:10.237100 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:08:10.237994 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:10.237963 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:08:11.173481 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:11.173441 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerID="6b5e16f179411429ca145175702749280acc4348b0ee2d4b85ec7e8ec047149f" exitCode=0 Jul 11 04:08:11.173648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:11.173521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerDied","Data":"6b5e16f179411429ca145175702749280acc4348b0ee2d4b85ec7e8ec047149f"} Jul 11 04:08:11.173648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:11.173610 2565 scope.go:117] "RemoveContainer" containerID="1ad34b6566cc43d34ee4d6faf2894eef8a0acbb1b97a4398ec78e218530a8045" Jul 11 04:08:11.186111 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:11.186080 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:08:12.310275 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:12.310250 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:08:12.374347 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:12.374309 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca\") pod \"a2c744b8-97c0-46cc-a378-a888f131e23f\" (UID: \"a2c744b8-97c0-46cc-a378-a888f131e23f\") " Jul 11 04:08:12.374729 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:12.374705 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2c744b8-97c0-46cc-a378-a888f131e23f" (UID: "a2c744b8-97c0-46cc-a378-a888f131e23f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:08:12.475657 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:12.475623 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c744b8-97c0-46cc-a378-a888f131e23f-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:08:13.185100 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:13.185071 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" Jul 11 04:08:13.185298 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:13.185108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn" event={"ID":"a2c744b8-97c0-46cc-a378-a888f131e23f","Type":"ContainerDied","Data":"5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3"} Jul 11 04:08:13.185298 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:13.185155 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca55bc84c5cf432597185aa28bec4835850063e90481f1247e4b97cdccfb2b3" Jul 11 04:08:13.213810 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:13.213776 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:08:13.219430 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:13.219403 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-rpzdn"] Jul 11 04:08:14.425746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:14.425705 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" path="/var/lib/kubelet/pods/a2c744b8-97c0-46cc-a378-a888f131e23f/volumes" Jul 11 04:08:44.763482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763444 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25"] Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763793 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763805 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763818 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763824 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763832 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763837 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763900 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.763953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.763908 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:08:44.768031 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.768010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:44.770265 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.770237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:08:44.773148 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.773119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25"] Jul 11 04:08:44.880074 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.880031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25\" (UID: \"639e63ab-a8d6-4b41-b26c-fb353c99e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:44.981127 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.981068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25\" (UID: \"639e63ab-a8d6-4b41-b26c-fb353c99e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:44.981739 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:44.981715 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25\" (UID: \"639e63ab-a8d6-4b41-b26c-fb353c99e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:45.079946 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:45.079850 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:45.219304 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:45.219276 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25"] Jul 11 04:08:45.221534 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:08:45.221503 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639e63ab_a8d6_4b41_b26c_fb353c99e22d.slice/crio-640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0 WatchSource:0}: Error finding container 640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0: Status 404 returned error can't find the container with id 640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0 Jul 11 04:08:45.331320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:45.331237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" event={"ID":"639e63ab-a8d6-4b41-b26c-fb353c99e22d","Type":"ContainerStarted","Data":"640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0"} Jul 11 04:08:46.337640 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:46.337549 2565 generic.go:356] "Generic (PLEG): container finished" podID="639e63ab-a8d6-4b41-b26c-fb353c99e22d" containerID="50514d1b1c8cb57ee8400953ebfc70071fd409cde41c1f21d280347645002db1" exitCode=0 Jul 11 04:08:46.337640 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:46.337603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" event={"ID":"639e63ab-a8d6-4b41-b26c-fb353c99e22d","Type":"ContainerDied","Data":"50514d1b1c8cb57ee8400953ebfc70071fd409cde41c1f21d280347645002db1"} Jul 11 04:08:47.477178 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:47.477151 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:47.605735 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:47.605602 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca\") pod \"639e63ab-a8d6-4b41-b26c-fb353c99e22d\" (UID: \"639e63ab-a8d6-4b41-b26c-fb353c99e22d\") " Jul 11 04:08:47.606026 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:47.605995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "639e63ab-a8d6-4b41-b26c-fb353c99e22d" (UID: "639e63ab-a8d6-4b41-b26c-fb353c99e22d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:08:47.706938 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:47.706897 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639e63ab-a8d6-4b41-b26c-fb353c99e22d-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:08:48.349063 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:48.349026 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" Jul 11 04:08:48.349229 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:48.349032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25" event={"ID":"639e63ab-a8d6-4b41-b26c-fb353c99e22d","Type":"ContainerDied","Data":"640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0"} Jul 11 04:08:48.349229 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:48.349184 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640d1ada9c144fd9d21c3dd6fe8ccf7b977613602b9a42afe186e8590cd585a0" Jul 11 04:08:54.838405 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:54.838367 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:08:54.838811 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:54.838591 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" containerID="cri-o://e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51" gracePeriod=30 Jul 11 04:08:55.239628 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:55.239583 2565 patch_prober.go:28] interesting pod/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:08:55.239821 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:08:55.239654 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:09:00.090878 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.090848 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:09:00.118730 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.118620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.118730 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.118671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.118943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.118836 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119049 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.118978 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:09:00.119131 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119131 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119104 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119131 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119060 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:09:00.119293 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119132 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119293 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119156 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119293 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119200 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119293 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119245 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnfgs\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs\") pod \"1b835bc0-6589-41b6-83b3-26626855ab44\" (UID: \"1b835bc0-6589-41b6-83b3-26626855ab44\") " Jul 11 04:09:00.119480 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119303 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:09:00.119595 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119573 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.119667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119604 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.119667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119620 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.119667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:09:00.119892 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.119868 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:09:00.121927 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.121888 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:09:00.122126 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.122066 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:09:00.122284 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.122247 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:09:00.122370 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.122333 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs" (OuterVolumeSpecName: "kube-api-access-nnfgs") pod "1b835bc0-6589-41b6-83b3-26626855ab44" (UID: "1b835bc0-6589-41b6-83b3-26626855ab44"). InnerVolumeSpecName "kube-api-access-nnfgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:09:00.220900 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220872 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.220900 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220897 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b835bc0-6589-41b6-83b3-26626855ab44-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.220900 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220907 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b835bc0-6589-41b6-83b3-26626855ab44-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.221113 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220915 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.221113 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220923 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfgs\" (UniqueName: \"kubernetes.io/projected/1b835bc0-6589-41b6-83b3-26626855ab44-kube-api-access-nnfgs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.221113 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.220932 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b835bc0-6589-41b6-83b3-26626855ab44-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:00.405779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.405745 2565 generic.go:356] "Generic (PLEG): container finished" podID="1b835bc0-6589-41b6-83b3-26626855ab44" containerID="e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51" exitCode=0 Jul 11 04:09:00.405996 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.405826 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" Jul 11 04:09:00.405996 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.405832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerDied","Data":"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51"} Jul 11 04:09:00.405996 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.405978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb" event={"ID":"1b835bc0-6589-41b6-83b3-26626855ab44","Type":"ContainerDied","Data":"548cf0c05bf14fffb5618f65d8c354359dddf66b10fc16bf221ecdc0566ca4f2"} Jul 11 04:09:00.406186 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.406015 2565 scope.go:117] "RemoveContainer" containerID="e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51" Jul 11 04:09:00.416102 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.416080 2565 scope.go:117] "RemoveContainer" containerID="924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca" Jul 11 04:09:00.424987 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.424962 2565 scope.go:117] "RemoveContainer" containerID="e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51" Jul 11 04:09:00.425441 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:09:00.425414 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51\": container with ID starting with e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51 not found: ID does not exist" containerID="e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51" Jul 11 04:09:00.425552 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.425455 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51"} err="failed to get container status \"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51\": rpc error: code = NotFound desc = could not find container \"e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51\": container with ID starting with e8c6482c89bf49522318ed1ffe44e16586df9daa8175384149b247e35d4f1c51 not found: ID does not exist" Jul 11 04:09:00.425552 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.425483 2565 scope.go:117] "RemoveContainer" containerID="924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca" Jul 11 04:09:00.425826 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:09:00.425798 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca\": container with ID starting with 924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca not found: ID does not exist" containerID="924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca" Jul 11 04:09:00.425910 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.425836 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca"} err="failed to get container status \"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca\": rpc error: code = NotFound desc = could not find container \"924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca\": container with ID starting with 924d10032f24dcbe8058f3fcfc9a7c18ee276974dea5269bc029c16c80faaeca not found: ID does not exist" Jul 11 04:09:00.427139 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.427117 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:09:00.430098 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:00.430077 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4b08374c-gw-openshift-default-bb7d6496c-jqsxb"] Jul 11 04:09:01.048316 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048275 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:09:01.048847 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048829 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:01.048906 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048852 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:01.048906 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048881 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639e63ab-a8d6-4b41-b26c-fb353c99e22d" containerName="revoke-keys" Jul 11 04:09:01.048906 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048890 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="639e63ab-a8d6-4b41-b26c-fb353c99e22d" containerName="revoke-keys" Jul 11 04:09:01.049009 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048972 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:01.049009 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048983 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c744b8-97c0-46cc-a378-a888f131e23f" containerName="revoke-keys" Jul 11 04:09:01.049009 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.048993 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="639e63ab-a8d6-4b41-b26c-fb353c99e22d" containerName="revoke-keys" Jul 11 04:09:01.053892 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.053868 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.056355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.056215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c784f07-openshift-default-dockercfg-tjs2w\"" Jul 11 04:09:01.064581 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.064514 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:09:01.130648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfhnk\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130910 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.131115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.130988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.231808 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfhnk\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.231964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232041 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232551 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232473 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.232632 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.232613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.234934 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.234909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.235120 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.235092 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.240506 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.240488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.240635 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.240614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfhnk\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk\") pod \"e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.368465 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.368358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:01.526782 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.526756 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:09:01.529518 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:09:01.529485 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f096ee_176e_4045_b890_33d6bb1cacda.slice/crio-6011601bd8d6ad896bcc5c3650aabfb509c204d0b6395f3f766fcc5f6e147bb8 WatchSource:0}: Error finding container 6011601bd8d6ad896bcc5c3650aabfb509c204d0b6395f3f766fcc5f6e147bb8: Status 404 returned error can't find the container with id 6011601bd8d6ad896bcc5c3650aabfb509c204d0b6395f3f766fcc5f6e147bb8 Jul 11 04:09:01.535128 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.535079 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:01.535252 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.535187 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:01.535252 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:01.535235 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:02.427546 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:02.427515 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" path="/var/lib/kubelet/pods/1b835bc0-6589-41b6-83b3-26626855ab44/volumes" Jul 11 04:09:02.427955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:02.427860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerStarted","Data":"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b"} Jul 11 04:09:02.427955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:02.427907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerStarted","Data":"6011601bd8d6ad896bcc5c3650aabfb509c204d0b6395f3f766fcc5f6e147bb8"} Jul 11 04:09:02.443607 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:02.443540 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podStartSLOduration=1.443521139 podStartE2EDuration="1.443521139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:09:02.442716122 +0000 UTC m=+1580.516332405" watchObservedRunningTime="2026-07-11 04:09:02.443521139 +0000 UTC m=+1580.517137399" Jul 11 04:09:03.369129 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:03.369092 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:03.373930 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:03.373901 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:03.427094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:03.427058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:03.428218 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:03.428193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:07.293092 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.293058 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:07.293454 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.293393 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:07.293454 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.293404 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:07.293533 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.293463 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b835bc0-6589-41b6-83b3-26626855ab44" containerName="istio-proxy" Jul 11 04:09:07.297494 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.297475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:07.299587 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.299565 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:09:07.303273 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.303248 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:07.394928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.394893 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r\" (UID: \"2b79ae08-84c0-41cc-ad06-0cc18096551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:07.495979 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.495935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r\" (UID: \"2b79ae08-84c0-41cc-ad06-0cc18096551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:07.496549 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.496530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r\" (UID: \"2b79ae08-84c0-41cc-ad06-0cc18096551a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:07.610450 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.610367 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:07.746157 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:07.746128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:07.748111 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:09:07.748086 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b79ae08_84c0_41cc_ad06_0cc18096551a.slice/crio-1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467 WatchSource:0}: Error finding container 1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467: Status 404 returned error can't find the container with id 1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467 Jul 11 04:09:08.448959 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:08.448926 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerID="9f027c01a4b15739b281a2f04b6542c205fc418bf5d8771c3a1c4c7508e58d54" exitCode=7 Jul 11 04:09:08.449357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:08.449009 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" event={"ID":"2b79ae08-84c0-41cc-ad06-0cc18096551a","Type":"ContainerDied","Data":"9f027c01a4b15739b281a2f04b6542c205fc418bf5d8771c3a1c4c7508e58d54"} Jul 11 04:09:08.449357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:08.449064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" event={"ID":"2b79ae08-84c0-41cc-ad06-0cc18096551a","Type":"ContainerStarted","Data":"1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467"} Jul 11 04:09:08.449357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:08.449163 2565 scope.go:117] "RemoveContainer" containerID="9f027c01a4b15739b281a2f04b6542c205fc418bf5d8771c3a1c4c7508e58d54" Jul 11 04:09:09.455417 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:09.455335 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerID="5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0" exitCode=7 Jul 11 04:09:09.455417 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:09.455406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" event={"ID":"2b79ae08-84c0-41cc-ad06-0cc18096551a","Type":"ContainerDied","Data":"5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0"} Jul 11 04:09:09.455856 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:09.455448 2565 scope.go:117] "RemoveContainer" containerID="9f027c01a4b15739b281a2f04b6542c205fc418bf5d8771c3a1c4c7508e58d54" Jul 11 04:09:09.455856 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:09.455634 2565 scope.go:117] "RemoveContainer" containerID="5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0" Jul 11 04:09:09.455936 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:09:09.455874 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r_odh-ai-gateway-infra(2b79ae08-84c0-41cc-ad06-0cc18096551a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" Jul 11 04:09:10.463119 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:10.463088 2565 scope.go:117] "RemoveContainer" containerID="5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0" Jul 11 04:09:10.463500 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:09:10.463278 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r_odh-ai-gateway-infra(2b79ae08-84c0-41cc-ad06-0cc18096551a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" Jul 11 04:09:17.498081 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.498042 2565 generic.go:356] "Generic (PLEG): container finished" podID="37f096ee-176e-4045-b890-33d6bb1cacda" containerID="2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b" exitCode=0 Jul 11 04:09:17.498547 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.498097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerDied","Data":"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b"} Jul 11 04:09:17.498547 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.498482 2565 scope.go:117] "RemoveContainer" containerID="2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b" Jul 11 04:09:17.499500 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.499467 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:17.499600 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.499550 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:17.499658 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:17.499596 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:09:18.427880 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:18.427852 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:18.504559 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:18.504527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerStarted","Data":"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c"} Jul 11 04:09:19.368898 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:19.368854 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:19.370078 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:19.370045 2565 patch_prober.go:28] interesting pod/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 11 04:09:19.370211 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:19.370110 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 11 04:09:20.369821 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:20.369785 2565 patch_prober.go:28] interesting pod/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 11 04:09:20.370228 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:20.369843 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 11 04:09:21.368914 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:21.368874 2565 patch_prober.go:28] interesting pod/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 11 04:09:21.369090 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:21.368934 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 11 04:09:22.372586 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:22.372556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:22.373077 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:22.372931 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:22.373554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:22.373533 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:09:25.420898 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:25.420863 2565 scope.go:117] "RemoveContainer" containerID="5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0" Jul 11 04:09:26.542647 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:26.542556 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerID="11fc61ce1c9ab2cc3a8dafd21f23db8344b020478ad940d43d17ee935b68c7a6" exitCode=0 Jul 11 04:09:26.542647 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:26.542625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" event={"ID":"2b79ae08-84c0-41cc-ad06-0cc18096551a","Type":"ContainerDied","Data":"11fc61ce1c9ab2cc3a8dafd21f23db8344b020478ad940d43d17ee935b68c7a6"} Jul 11 04:09:26.543098 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:26.542706 2565 scope.go:117] "RemoveContainer" containerID="5db5d8052229cbaf37a71f73c81fbb5eef297824aab58275cefe7f3e7e9aaff0" Jul 11 04:09:27.568401 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:27.568363 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:27.675817 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:27.675795 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:27.694763 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:27.694738 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca\") pod \"2b79ae08-84c0-41cc-ad06-0cc18096551a\" (UID: \"2b79ae08-84c0-41cc-ad06-0cc18096551a\") " Jul 11 04:09:27.695077 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:27.695055 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b79ae08-84c0-41cc-ad06-0cc18096551a" (UID: "2b79ae08-84c0-41cc-ad06-0cc18096551a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:09:27.796108 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:27.796072 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b79ae08-84c0-41cc-ad06-0cc18096551a-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:09:28.553749 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:28.553723 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" Jul 11 04:09:28.553913 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:28.553716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r" event={"ID":"2b79ae08-84c0-41cc-ad06-0cc18096551a","Type":"ContainerDied","Data":"1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467"} Jul 11 04:09:28.553913 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:28.553847 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd5366df3aefee6b722a7b765ac27b89680cb603555ce0294b53bbb3fde7467" Jul 11 04:09:28.580801 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:28.580765 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:28.584670 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:28.584645 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-hfw4r"] Jul 11 04:09:30.426257 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:09:30.426224 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" path="/var/lib/kubelet/pods/2b79ae08-84c0-41cc-ad06-0cc18096551a/volumes" Jul 11 04:10:07.335737 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.335695 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr"] Jul 11 04:10:07.336165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336072 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336084 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336094 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336099 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336165 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336175 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.336320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.336180 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:07.340585 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.340569 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:07.342844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.342817 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:10:07.346107 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.346082 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr"] Jul 11 04:10:07.371664 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.371632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr\" (UID: \"c77394c3-1ac7-4a3a-9431-6072d918b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:07.472794 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.472736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr\" (UID: \"c77394c3-1ac7-4a3a-9431-6072d918b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:07.473429 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.473409 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr\" (UID: \"c77394c3-1ac7-4a3a-9431-6072d918b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:07.652162 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.652128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:07.787885 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:07.787846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr"] Jul 11 04:10:07.790849 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:10:07.790814 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77394c3_1ac7_4a3a_9431_6072d918b0ba.slice/crio-c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416 WatchSource:0}: Error finding container c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416: Status 404 returned error can't find the container with id c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416 Jul 11 04:10:08.732695 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:08.732641 2565 generic.go:356] "Generic (PLEG): container finished" podID="c77394c3-1ac7-4a3a-9431-6072d918b0ba" containerID="de80ec6f9d3613ca2ece48426406dddc26137b95b231a013b73a64065bca903b" exitCode=0 Jul 11 04:10:08.733085 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:08.732713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" event={"ID":"c77394c3-1ac7-4a3a-9431-6072d918b0ba","Type":"ContainerDied","Data":"de80ec6f9d3613ca2ece48426406dddc26137b95b231a013b73a64065bca903b"} Jul 11 04:10:08.733085 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:08.732769 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" event={"ID":"c77394c3-1ac7-4a3a-9431-6072d918b0ba","Type":"ContainerStarted","Data":"c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416"} Jul 11 04:10:09.873551 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:09.873525 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:09.895027 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:09.894999 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca\") pod \"c77394c3-1ac7-4a3a-9431-6072d918b0ba\" (UID: \"c77394c3-1ac7-4a3a-9431-6072d918b0ba\") " Jul 11 04:10:09.895330 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:09.895309 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c77394c3-1ac7-4a3a-9431-6072d918b0ba" (UID: "c77394c3-1ac7-4a3a-9431-6072d918b0ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:10:09.996750 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:09.996632 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77394c3-1ac7-4a3a-9431-6072d918b0ba-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:10.743047 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:10.742972 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" Jul 11 04:10:10.743191 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:10.742971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr" event={"ID":"c77394c3-1ac7-4a3a-9431-6072d918b0ba","Type":"ContainerDied","Data":"c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416"} Jul 11 04:10:10.743191 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:10.743089 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c694bf72db03ee66f77ce0c05a3d3173aa5c733add15a0a2fd78da048accd416" Jul 11 04:10:33.420597 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.420562 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:10:33.421094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.420807 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" containerID="cri-o://5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c" gracePeriod=30 Jul 11 04:10:33.720629 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.720527 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:10:33.721137 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.721071 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:33.721137 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.721096 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b79ae08-84c0-41cc-ad06-0cc18096551a" containerName="revoke-keys" Jul 11 04:10:33.721137 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.721131 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77394c3-1ac7-4a3a-9431-6072d918b0ba" containerName="revoke-keys" Jul 11 04:10:33.721137 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.721140 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77394c3-1ac7-4a3a-9431-6072d918b0ba" containerName="revoke-keys" Jul 11 04:10:33.721387 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.721215 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77394c3-1ac7-4a3a-9431-6072d918b0ba" containerName="revoke-keys" Jul 11 04:10:33.733611 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.732953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.735702 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.735654 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:10:33.736421 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.736394 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-55671289-openshift-default-dockercfg-6dvkq\"" Jul 11 04:10:33.815165 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpc5h\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815492 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815492 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815492 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.815492 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.815479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916210 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916210 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpc5h\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.916471 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.917412 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.917412 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.917412 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.916982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.917412 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.917017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.917412 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.917258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.918817 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.918796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.919034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.919018 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.924159 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.924142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:33.924522 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:33.924501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpc5h\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h\") pod \"e2e-derive-55671289-openshift-default-857449699f-4d5lv\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:34.045259 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.045171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:34.183210 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.183181 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:10:34.184668 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:10:34.184642 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a73d07_e49f_4f01_9b58_c98a5c51b81a.slice/crio-d039f4734e9e71f30c542276123a3cfb37f9e6dadf27a46f8632986d71632f15 WatchSource:0}: Error finding container d039f4734e9e71f30c542276123a3cfb37f9e6dadf27a46f8632986d71632f15: Status 404 returned error can't find the container with id d039f4734e9e71f30c542276123a3cfb37f9e6dadf27a46f8632986d71632f15 Jul 11 04:10:34.186957 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.186925 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:34.187016 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.186997 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:34.187099 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.187044 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:34.860327 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.860294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerStarted","Data":"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984"} Jul 11 04:10:34.860792 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.860354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerStarted","Data":"d039f4734e9e71f30c542276123a3cfb37f9e6dadf27a46f8632986d71632f15"} Jul 11 04:10:34.885119 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:34.885060 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podStartSLOduration=1.8850420730000002 podStartE2EDuration="1.885042073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:10:34.884275099 +0000 UTC m=+1672.957891378" watchObservedRunningTime="2026-07-11 04:10:34.885042073 +0000 UTC m=+1672.958658332" Jul 11 04:10:35.046015 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:35.045969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:35.070045 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:35.069996 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:35.865706 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:35.865649 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:35.866776 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:35.866756 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:37.376074 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:37.376040 2565 patch_prober.go:28] interesting pod/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:10:37.376479 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:37.376096 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:10:38.707557 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.707530 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:10:38.870663 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870663 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870670 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870730 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870758 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870812 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.870943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.870841 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfhnk\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.871184 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.871184 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871150 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:10:38.871277 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871200 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data\") pod \"37f096ee-176e-4045-b890-33d6bb1cacda\" (UID: \"37f096ee-176e-4045-b890-33d6bb1cacda\") " Jul 11 04:10:38.871337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871285 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:10:38.871337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:10:38.871571 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871515 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.871571 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871533 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.871571 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871542 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.871839 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871720 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:10:38.871986 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.871966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data" (OuterVolumeSpecName: "istio-data") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:10:38.873729 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.873652 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:10:38.874126 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.873982 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk" (OuterVolumeSpecName: "kube-api-access-sfhnk") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "kube-api-access-sfhnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:10:38.874126 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.873995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token" (OuterVolumeSpecName: "istio-token") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:10:38.874388 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.873886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37f096ee-176e-4045-b890-33d6bb1cacda" (UID: "37f096ee-176e-4045-b890-33d6bb1cacda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:10:38.886826 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.886797 2565 generic.go:356] "Generic (PLEG): container finished" podID="37f096ee-176e-4045-b890-33d6bb1cacda" containerID="5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c" exitCode=0 Jul 11 04:10:38.886985 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.886873 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" Jul 11 04:10:38.886985 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.886877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerDied","Data":"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c"} Jul 11 04:10:38.886985 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.886976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv" event={"ID":"37f096ee-176e-4045-b890-33d6bb1cacda","Type":"ContainerDied","Data":"6011601bd8d6ad896bcc5c3650aabfb509c204d0b6395f3f766fcc5f6e147bb8"} Jul 11 04:10:38.887162 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.887008 2565 scope.go:117] "RemoveContainer" containerID="5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c" Jul 11 04:10:38.896625 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.896607 2565 scope.go:117] "RemoveContainer" containerID="2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b" Jul 11 04:10:38.905002 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.904979 2565 scope.go:117] "RemoveContainer" containerID="5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c" Jul 11 04:10:38.905269 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:10:38.905248 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c\": container with ID starting with 5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c not found: ID does not exist" containerID="5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c" Jul 11 04:10:38.905337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.905277 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c"} err="failed to get container status \"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c\": rpc error: code = NotFound desc = could not find container \"5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c\": container with ID starting with 5c6cc7f30a79cfb0bed122b87ab35c9a7c358385c06a1adb99087487273c7d2c not found: ID does not exist" Jul 11 04:10:38.905337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.905295 2565 scope.go:117] "RemoveContainer" containerID="2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b" Jul 11 04:10:38.905528 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:10:38.905512 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b\": container with ID starting with 2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b not found: ID does not exist" containerID="2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b" Jul 11 04:10:38.905569 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.905534 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b"} err="failed to get container status \"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b\": rpc error: code = NotFound desc = could not find container \"2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b\": container with ID starting with 2d728ec74eed44d8a119ebc37992a840993ede071a1c84fa1227f72721c9529b not found: ID does not exist" Jul 11 04:10:38.908764 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.908743 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:10:38.912114 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.912093 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c784f07-openshift-default-bdbfbf767-jwpsv"] Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.972951 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f096ee-176e-4045-b890-33d6bb1cacda-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.972981 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f096ee-176e-4045-b890-33d6bb1cacda-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.972992 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfhnk\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-kube-api-access-sfhnk\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.973002 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f096ee-176e-4045-b890-33d6bb1cacda-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.973012 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:38.973034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:38.973021 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f096ee-176e-4045-b890-33d6bb1cacda-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:10:39.250025 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.249920 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:10:39.250536 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.250508 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:10:39.250536 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.250534 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:10:39.250745 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.250551 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:10:39.250745 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.250559 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:10:39.250745 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.250648 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:10:39.255830 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.255804 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:10:39.258314 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.258287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:10:39.260790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.260645 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:10:39.378045 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.377983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-9kwfn\" (UID: \"3238d230-867f-4ac6-98c1-c16e0881d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:10:39.479537 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.479498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-9kwfn\" (UID: \"3238d230-867f-4ac6-98c1-c16e0881d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:10:39.480216 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.480189 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-9kwfn\" (UID: \"3238d230-867f-4ac6-98c1-c16e0881d2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:10:39.573192 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.573092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:10:39.731992 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.731835 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:10:39.892377 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:39.892343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerStarted","Data":"21058635ec7c2370138fc704826f97ce0262f093141e6fbc9c53f889426f6be8"} Jul 11 04:10:40.425779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:40.425695 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" path="/var/lib/kubelet/pods/37f096ee-176e-4045-b890-33d6bb1cacda/volumes" Jul 11 04:10:40.900065 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:40.900033 2565 generic.go:356] "Generic (PLEG): container finished" podID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerID="3d11c28b464e7c945ec1fd0a0885ee3d6d90cbcb86be0976a13a409b96c7f7b0" exitCode=7 Jul 11 04:10:40.900470 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:40.900118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerDied","Data":"3d11c28b464e7c945ec1fd0a0885ee3d6d90cbcb86be0976a13a409b96c7f7b0"} Jul 11 04:10:40.900470 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:40.900301 2565 scope.go:117] "RemoveContainer" containerID="3d11c28b464e7c945ec1fd0a0885ee3d6d90cbcb86be0976a13a409b96c7f7b0" Jul 11 04:10:41.907410 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:41.907374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerStarted","Data":"632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e"} Jul 11 04:10:41.923734 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:41.923657 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" podStartSLOduration=2.923640353 podStartE2EDuration="2.923640353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:10:41.922333631 +0000 UTC m=+1679.995949890" watchObservedRunningTime="2026-07-11 04:10:41.923640353 +0000 UTC m=+1679.997256655" Jul 11 04:10:42.917273 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:42.917236 2565 generic.go:356] "Generic (PLEG): container finished" podID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerID="632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e" exitCode=7 Jul 11 04:10:42.917738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:42.917307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerDied","Data":"632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e"} Jul 11 04:10:42.917738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:42.917363 2565 scope.go:117] "RemoveContainer" containerID="3d11c28b464e7c945ec1fd0a0885ee3d6d90cbcb86be0976a13a409b96c7f7b0" Jul 11 04:10:42.917738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:42.917604 2565 scope.go:117] "RemoveContainer" containerID="632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e" Jul 11 04:10:42.917879 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:10:42.917846 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-55671289-9kwfn_odh-ai-gateway-infra(3238d230-867f-4ac6-98c1-c16e0881d2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" Jul 11 04:10:43.926725 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:43.926698 2565 scope.go:117] "RemoveContainer" containerID="632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e" Jul 11 04:10:43.927098 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:10:43.926906 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-55671289-9kwfn_odh-ai-gateway-infra(3238d230-867f-4ac6-98c1-c16e0881d2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" Jul 11 04:10:45.938374 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.938337 2565 generic.go:356] "Generic (PLEG): container finished" podID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerID="80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984" exitCode=0 Jul 11 04:10:45.938919 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.938395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerDied","Data":"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984"} Jul 11 04:10:45.938919 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.938904 2565 scope.go:117] "RemoveContainer" containerID="80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984" Jul 11 04:10:45.939990 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.939960 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:45.940057 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.940027 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:45.940096 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:45.940063 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:10:46.948160 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:46.948115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerStarted","Data":"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8"} Jul 11 04:10:47.046749 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:47.046715 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:47.047126 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:47.047101 2565 patch_prober.go:28] interesting pod/e2e-derive-55671289-openshift-default-857449699f-4d5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 11 04:10:47.047320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:47.047294 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 11 04:10:48.046749 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:48.046701 2565 patch_prober.go:28] interesting pod/e2e-derive-55671289-openshift-default-857449699f-4d5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 11 04:10:48.047287 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:48.046791 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 11 04:10:49.045981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:49.045939 2565 patch_prober.go:28] interesting pod/e2e-derive-55671289-openshift-default-857449699f-4d5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 11 04:10:49.046301 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:49.046271 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 11 04:10:50.046434 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:50.046379 2565 patch_prober.go:28] interesting pod/e2e-derive-55671289-openshift-default-857449699f-4d5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 11 04:10:50.046976 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:50.046455 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 11 04:10:50.865834 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:50.865800 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:51.049324 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:51.049288 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:51.050349 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:51.050327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:10:59.421198 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:10:59.421166 2565 scope.go:117] "RemoveContainer" containerID="632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e" Jul 11 04:11:00.013959 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:00.013871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerStarted","Data":"6403256c1791bacb09a7add3a3a861f6923823010a13edada21902305ae98c86"} Jul 11 04:11:01.020529 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:01.020494 2565 generic.go:356] "Generic (PLEG): container finished" podID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerID="6403256c1791bacb09a7add3a3a861f6923823010a13edada21902305ae98c86" exitCode=0 Jul 11 04:11:01.020987 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:01.020570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerDied","Data":"6403256c1791bacb09a7add3a3a861f6923823010a13edada21902305ae98c86"} Jul 11 04:11:01.020987 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:01.020630 2565 scope.go:117] "RemoveContainer" containerID="632c7c56ca1cbd42152606df47159fee7fa4b0087a695efc998a68cce2f37c3e" Jul 11 04:11:01.036310 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:01.036287 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:11:02.167432 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:02.167406 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:11:02.296203 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:02.296108 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca\") pod \"3238d230-867f-4ac6-98c1-c16e0881d2f2\" (UID: \"3238d230-867f-4ac6-98c1-c16e0881d2f2\") " Jul 11 04:11:02.296490 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:02.296465 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3238d230-867f-4ac6-98c1-c16e0881d2f2" (UID: "3238d230-867f-4ac6-98c1-c16e0881d2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:11:02.396974 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:02.396932 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238d230-867f-4ac6-98c1-c16e0881d2f2-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:11:03.033045 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:03.032957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" event={"ID":"3238d230-867f-4ac6-98c1-c16e0881d2f2","Type":"ContainerDied","Data":"21058635ec7c2370138fc704826f97ce0262f093141e6fbc9c53f889426f6be8"} Jul 11 04:11:03.033045 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:03.033031 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21058635ec7c2370138fc704826f97ce0262f093141e6fbc9c53f889426f6be8" Jul 11 04:11:03.033045 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:03.032970 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn" Jul 11 04:11:03.057960 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:03.057927 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:11:03.063572 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:03.063543 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-9kwfn"] Jul 11 04:11:04.425837 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:04.425797 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" path="/var/lib/kubelet/pods/3238d230-867f-4ac6-98c1-c16e0881d2f2/volumes" Jul 11 04:11:39.288088 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288048 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87"] Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288387 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288398 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288416 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288422 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288475 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f096ee-176e-4045-b890-33d6bb1cacda" containerName="istio-proxy" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288487 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288492 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.288554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.288497 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:11:39.291601 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.291583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:39.293818 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.293792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:11:39.297666 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.297638 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87"] Jul 11 04:11:39.332201 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.332165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-lxr87\" (UID: \"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:39.433328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.433291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-lxr87\" (UID: \"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:39.433992 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.433973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55671289-lxr87\" (UID: \"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:39.603510 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.603414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:39.735821 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:39.735793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87"] Jul 11 04:11:39.737782 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:11:39.737749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f59139_3ea3_48b6_aaeb_9da2ccf2dea1.slice/crio-33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e WatchSource:0}: Error finding container 33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e: Status 404 returned error can't find the container with id 33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e Jul 11 04:11:40.219159 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:40.219054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" event={"ID":"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1","Type":"ContainerStarted","Data":"55aaf75783ca5372458e1e181b2444fb8fe7171a3cd7a5c2916cab875b0ada4c"} Jul 11 04:11:40.219159 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:40.219110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" event={"ID":"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1","Type":"ContainerStarted","Data":"33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e"} Jul 11 04:11:40.235312 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:40.235253 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" podStartSLOduration=1.23523828 podStartE2EDuration="1.23523828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:11:40.234502644 +0000 UTC m=+1738.308118904" watchObservedRunningTime="2026-07-11 04:11:40.23523828 +0000 UTC m=+1738.308854540" Jul 11 04:11:41.225223 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:41.225187 2565 generic.go:356] "Generic (PLEG): container finished" podID="d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1" containerID="55aaf75783ca5372458e1e181b2444fb8fe7171a3cd7a5c2916cab875b0ada4c" exitCode=0 Jul 11 04:11:41.225606 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:41.225264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" event={"ID":"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1","Type":"ContainerDied","Data":"55aaf75783ca5372458e1e181b2444fb8fe7171a3cd7a5c2916cab875b0ada4c"} Jul 11 04:11:42.362436 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:42.362408 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:42.460428 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:42.460384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca\") pod \"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1\" (UID: \"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1\") " Jul 11 04:11:42.460836 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:42.460815 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1" (UID: "d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:11:42.561419 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:42.561346 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:11:43.236350 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:43.236317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" event={"ID":"d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1","Type":"ContainerDied","Data":"33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e"} Jul 11 04:11:43.236350 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:43.236350 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55671289-lxr87" Jul 11 04:11:43.236562 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:11:43.236374 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b1abe2652b359e980b64bf09e541f8d19c9257090c986f4d38a88a51a1614e" Jul 11 04:12:04.533365 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:04.533327 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:12:04.533869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:04.533520 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" containerID="cri-o://e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8" gracePeriod=30 Jul 11 04:12:06.052089 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:06.052053 2565 patch_prober.go:28] interesting pod/e2e-derive-55671289-openshift-default-857449699f-4d5lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:12:06.052504 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:06.052124 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:12:09.796147 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.796122 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:12:09.914167 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914353 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914353 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914212 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914353 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914353 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914293 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914358 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914403 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpc5h\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h\") pod \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\" (UID: \"16a73d07-e49f-4f01-9b58-c98a5c51b81a\") " Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914506 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:12:09.914631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:12:09.914943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914625 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:12:09.914943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914823 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:12:09.914943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914849 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:09.914943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914873 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:09.914943 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.914887 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:09.915346 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.915204 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data" (OuterVolumeSpecName: "istio-data") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:12:09.916770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.916733 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:12:09.916884 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.916843 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token" (OuterVolumeSpecName: "istio-token") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:12:09.917128 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.917109 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h" (OuterVolumeSpecName: "kube-api-access-qpc5h") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "kube-api-access-qpc5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:12:09.917177 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:09.917120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16a73d07-e49f-4f01-9b58-c98a5c51b81a" (UID: "16a73d07-e49f-4f01-9b58-c98a5c51b81a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:12:10.015984 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.015937 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.015984 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.015974 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.015984 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.015987 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.015984 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.015998 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.016267 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.016010 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a73d07-e49f-4f01-9b58-c98a5c51b81a-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.016267 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.016021 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpc5h\" (UniqueName: \"kubernetes.io/projected/16a73d07-e49f-4f01-9b58-c98a5c51b81a-kube-api-access-qpc5h\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:12:10.358049 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.358002 2565 generic.go:356] "Generic (PLEG): container finished" podID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerID="e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8" exitCode=0 Jul 11 04:12:10.358333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.358058 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerDied","Data":"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8"} Jul 11 04:12:10.358333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.358081 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" Jul 11 04:12:10.358333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.358126 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv" event={"ID":"16a73d07-e49f-4f01-9b58-c98a5c51b81a","Type":"ContainerDied","Data":"d039f4734e9e71f30c542276123a3cfb37f9e6dadf27a46f8632986d71632f15"} Jul 11 04:12:10.358333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.358149 2565 scope.go:117] "RemoveContainer" containerID="e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8" Jul 11 04:12:10.367981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.367962 2565 scope.go:117] "RemoveContainer" containerID="80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984" Jul 11 04:12:10.375978 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.375959 2565 scope.go:117] "RemoveContainer" containerID="e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8" Jul 11 04:12:10.376200 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:12:10.376183 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8\": container with ID starting with e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8 not found: ID does not exist" containerID="e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8" Jul 11 04:12:10.376254 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.376208 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8"} err="failed to get container status \"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8\": rpc error: code = NotFound desc = could not find container \"e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8\": container with ID starting with e2467557c1d23bd6eaf3a1f264af55414b615d2c2c96387d6f43fd8a4049caf8 not found: ID does not exist" Jul 11 04:12:10.376254 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.376223 2565 scope.go:117] "RemoveContainer" containerID="80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984" Jul 11 04:12:10.376423 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:12:10.376409 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984\": container with ID starting with 80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984 not found: ID does not exist" containerID="80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984" Jul 11 04:12:10.376464 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.376427 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984"} err="failed to get container status \"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984\": rpc error: code = NotFound desc = could not find container \"80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984\": container with ID starting with 80a7bf25698e4c5f14650214d21fede68f17429c3488f5f08e3283183103d984 not found: ID does not exist" Jul 11 04:12:10.379157 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.379135 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:12:10.383067 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.383044 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-55671289-openshift-default-857449699f-4d5lv"] Jul 11 04:12:10.426075 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:10.426046 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" path="/var/lib/kubelet/pods/16a73d07-e49f-4f01-9b58-c98a5c51b81a/volumes" Jul 11 04:12:42.508170 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:42.508138 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:12:42.513215 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:12:42.513192 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:13:39.661332 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661292 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661714 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661730 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661739 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661744 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3238d230-867f-4ac6-98c1-c16e0881d2f2" containerName="revoke-keys" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661758 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661763 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661770 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1" containerName="revoke-keys" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661776 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1" containerName="revoke-keys" Jul 11 04:13:39.661844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661846 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1" containerName="revoke-keys" Jul 11 04:13:39.662242 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661854 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.662242 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.661862 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a73d07-e49f-4f01-9b58-c98a5c51b81a" containerName="istio-proxy" Jul 11 04:13:39.665037 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.665017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.667172 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.667151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d7da7c14-openshift-default-dockercfg-lfcch\"" Jul 11 04:13:39.676717 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.676669 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:13:39.820768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.820768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.820981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.820981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820899 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.820981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.820981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.821119 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.820994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.821119 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.821010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9sb\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.821119 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.821079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922123 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922123 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922210 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9sb\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922341 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922824 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922899 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922864 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.922953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.922879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.924630 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.924608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.925163 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.925142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.929950 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.929920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.930527 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.930500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9sb\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb\") pod \"e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:39.976750 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:39.976705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:40.135123 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.135095 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:13:40.137634 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:13:40.137601 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4a25ba_0155_456a_9227_9b0c9e201236.slice/crio-e6e46f3069c2cc190f51b3aa3c6f36ef09008e20e68a6bfa190dc72f234b0437 WatchSource:0}: Error finding container e6e46f3069c2cc190f51b3aa3c6f36ef09008e20e68a6bfa190dc72f234b0437: Status 404 returned error can't find the container with id e6e46f3069c2cc190f51b3aa3c6f36ef09008e20e68a6bfa190dc72f234b0437 Jul 11 04:13:40.139837 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.139804 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:40.139911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.139873 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:40.139911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.139901 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:40.768965 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.768916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerStarted","Data":"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62"} Jul 11 04:13:40.769384 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.768981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerStarted","Data":"e6e46f3069c2cc190f51b3aa3c6f36ef09008e20e68a6bfa190dc72f234b0437"} Jul 11 04:13:40.788247 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.788190 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podStartSLOduration=1.788171734 podStartE2EDuration="1.788171734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:13:40.786488595 +0000 UTC m=+1858.860104876" watchObservedRunningTime="2026-07-11 04:13:40.788171734 +0000 UTC m=+1858.861787993" Jul 11 04:13:40.977474 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:40.977436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:41.982243 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:41.982215 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:42.779363 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:42.779327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:42.780221 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:42.780192 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:53.838506 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.838413 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerID="1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62" exitCode=0 Jul 11 04:13:53.838506 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.838480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerDied","Data":"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62"} Jul 11 04:13:53.838953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.838891 2565 scope.go:117] "RemoveContainer" containerID="1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62" Jul 11 04:13:53.840091 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.840058 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:53.840173 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.840154 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:53.840230 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:53.840213 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:13:54.565022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.564969 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x"] Jul 11 04:13:54.569322 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.569298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:54.571510 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.571485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:13:54.576035 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.575974 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x"] Jul 11 04:13:54.655095 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.655055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x\" (UID: \"068e2f7b-9b93-4b51-988b-04f86cba3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:54.757762 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.756700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x\" (UID: \"068e2f7b-9b93-4b51-988b-04f86cba3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:54.757762 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.757705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x\" (UID: \"068e2f7b-9b93-4b51-988b-04f86cba3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:54.859566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.859448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerStarted","Data":"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc"} Jul 11 04:13:54.883513 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.883477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:54.977828 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.977801 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:54.978876 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.978850 2565 patch_prober.go:28] interesting pod/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 11 04:13:54.978971 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:54.978897 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 11 04:13:55.055705 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.043167 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x"] Jul 11 04:13:55.866961 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.866923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" event={"ID":"068e2f7b-9b93-4b51-988b-04f86cba3eb3","Type":"ContainerStarted","Data":"adf1f10d80ed62e8359eaa2f232808b394fda494c8878f1b75962070f3607ba2"} Jul 11 04:13:55.867370 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.866977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" event={"ID":"068e2f7b-9b93-4b51-988b-04f86cba3eb3","Type":"ContainerStarted","Data":"34caab020510bcdf1c1877e13527f392b3af92a94ae994b9afbaef265c37fe9f"} Jul 11 04:13:55.883810 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.883758 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" podStartSLOduration=1.88374415 podStartE2EDuration="1.88374415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:13:55.881555763 +0000 UTC m=+1873.955172041" watchObservedRunningTime="2026-07-11 04:13:55.88374415 +0000 UTC m=+1873.957360436" Jul 11 04:13:55.977936 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.977902 2565 patch_prober.go:28] interesting pod/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 11 04:13:55.978107 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:55.977968 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 11 04:13:56.876721 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:56.876667 2565 generic.go:356] "Generic (PLEG): container finished" podID="068e2f7b-9b93-4b51-988b-04f86cba3eb3" containerID="adf1f10d80ed62e8359eaa2f232808b394fda494c8878f1b75962070f3607ba2" exitCode=0 Jul 11 04:13:56.876721 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:56.876709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" event={"ID":"068e2f7b-9b93-4b51-988b-04f86cba3eb3","Type":"ContainerDied","Data":"adf1f10d80ed62e8359eaa2f232808b394fda494c8878f1b75962070f3607ba2"} Jul 11 04:13:56.977697 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:56.977640 2565 patch_prober.go:28] interesting pod/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 11 04:13:56.977849 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:56.977724 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 11 04:13:57.779803 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:57.779769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:57.977901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:57.977868 2565 patch_prober.go:28] interesting pod/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 11 04:13:57.978276 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:57.977926 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 11 04:13:58.022291 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.022269 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:58.091094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.091009 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca\") pod \"068e2f7b-9b93-4b51-988b-04f86cba3eb3\" (UID: \"068e2f7b-9b93-4b51-988b-04f86cba3eb3\") " Jul 11 04:13:58.091357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.091336 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "068e2f7b-9b93-4b51-988b-04f86cba3eb3" (UID: "068e2f7b-9b93-4b51-988b-04f86cba3eb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:13:58.192528 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.192488 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068e2f7b-9b93-4b51-988b-04f86cba3eb3-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:13:58.887556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.887524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" event={"ID":"068e2f7b-9b93-4b51-988b-04f86cba3eb3","Type":"ContainerDied","Data":"34caab020510bcdf1c1877e13527f392b3af92a94ae994b9afbaef265c37fe9f"} Jul 11 04:13:58.887556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.887554 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x" Jul 11 04:13:58.887799 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.887573 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34caab020510bcdf1c1877e13527f392b3af92a94ae994b9afbaef265c37fe9f" Jul 11 04:13:58.981744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.981715 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:13:58.982812 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:13:58.982786 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:14:20.343539 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:20.343446 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:14:20.344061 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:20.343663 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" containerID="cri-o://1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc" gracePeriod=30 Jul 11 04:14:25.602456 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.602221 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:14:25.753846 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.753799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.753858 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.753884 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t9sb\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.753914 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754003 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754028 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754054 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754053 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754386 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754386 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket\") pod \"0e4a25ba-0155-456a-9227-9b0c9e201236\" (UID: \"0e4a25ba-0155-456a-9227-9b0c9e201236\") " Jul 11 04:14:25.754386 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754162 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:14:25.754542 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754410 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:14:25.754542 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754432 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.754651 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754616 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:14:25.754770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.754746 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:14:25.755094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.755065 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:14:25.756666 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.756634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:14:25.756939 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.756898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:14:25.757032 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.756942 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb" (OuterVolumeSpecName: "kube-api-access-7t9sb") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "kube-api-access-7t9sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:14:25.757083 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.757051 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e4a25ba-0155-456a-9227-9b0c9e201236" (UID: "0e4a25ba-0155-456a-9227-9b0c9e201236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:14:25.855689 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855646 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855711 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855721 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855729 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e4a25ba-0155-456a-9227-9b0c9e201236-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855737 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855745 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e4a25ba-0155-456a-9227-9b0c9e201236-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855754 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9sb\" (UniqueName: \"kubernetes.io/projected/0e4a25ba-0155-456a-9227-9b0c9e201236-kube-api-access-7t9sb\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:25.855869 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:25.855761 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e4a25ba-0155-456a-9227-9b0c9e201236-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:14:26.017421 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.017332 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerID="1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc" exitCode=0 Jul 11 04:14:26.017421 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.017371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerDied","Data":"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc"} Jul 11 04:14:26.017421 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.017381 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" Jul 11 04:14:26.017652 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.017438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g" event={"ID":"0e4a25ba-0155-456a-9227-9b0c9e201236","Type":"ContainerDied","Data":"e6e46f3069c2cc190f51b3aa3c6f36ef09008e20e68a6bfa190dc72f234b0437"} Jul 11 04:14:26.017652 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.017461 2565 scope.go:117] "RemoveContainer" containerID="1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc" Jul 11 04:14:26.029953 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.029933 2565 scope.go:117] "RemoveContainer" containerID="1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62" Jul 11 04:14:26.038472 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.038453 2565 scope.go:117] "RemoveContainer" containerID="1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc" Jul 11 04:14:26.038758 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:14:26.038740 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc\": container with ID starting with 1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc not found: ID does not exist" containerID="1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc" Jul 11 04:14:26.038849 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.038770 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc"} err="failed to get container status \"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc\": rpc error: code = NotFound desc = could not find container \"1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc\": container with ID starting with 1bae14f42eb7150442021c0ece0cf352be9dfcc125476306c20661999146f7bc not found: ID does not exist" Jul 11 04:14:26.038849 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.038796 2565 scope.go:117] "RemoveContainer" containerID="1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62" Jul 11 04:14:26.039035 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:14:26.039014 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62\": container with ID starting with 1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62 not found: ID does not exist" containerID="1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62" Jul 11 04:14:26.039090 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.039042 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62"} err="failed to get container status \"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62\": rpc error: code = NotFound desc = could not find container \"1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62\": container with ID starting with 1fd07b317cbe38b7194c08c1c917eb94f7e95114160125fae53ca66441238d62 not found: ID does not exist" Jul 11 04:14:26.043186 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.043161 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:14:26.050021 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.049990 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d7da7c14-openshift-default-5988f56d47-rgd4g"] Jul 11 04:14:26.426262 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:26.426233 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" path="/var/lib/kubelet/pods/0e4a25ba-0155-456a-9227-9b0c9e201236/volumes" Jul 11 04:14:39.817458 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.817424 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:14:39.817967 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.817944 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818056 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.817971 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818056 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.817986 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068e2f7b-9b93-4b51-988b-04f86cba3eb3" containerName="revoke-keys" Jul 11 04:14:39.818056 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.817994 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="068e2f7b-9b93-4b51-988b-04f86cba3eb3" containerName="revoke-keys" Jul 11 04:14:39.818056 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.818019 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818056 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.818029 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.818118 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.818131 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4a25ba-0155-456a-9227-9b0c9e201236" containerName="istio-proxy" Jul 11 04:14:39.818355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.818141 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="068e2f7b-9b93-4b51-988b-04f86cba3eb3" containerName="revoke-keys" Jul 11 04:14:39.822810 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.822785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.824945 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.824922 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d9f87d6a-openshift-default-dockercfg-hr8st\"" Jul 11 04:14:39.833789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.833758 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:14:39.978631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978631 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grg8g\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.978917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:39.979140 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:39.978933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080272 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080272 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080272 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080277 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grg8g\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080896 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080818 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.080997 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.080972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.081215 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.081187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.081406 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.081383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.081623 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.081605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.083083 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.083060 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.083301 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.083280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.087829 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.087799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.088143 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.088123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grg8g\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g\") pod \"e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.135079 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.135043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:40.286059 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.286027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:14:40.288342 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:14:40.288318 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fbc239_afcb_48d9_aa89_78f602e5c949.slice/crio-a43af90643342adcc950114a2e8a0ae26c7a447dd5d64932a9e789340175e09a WatchSource:0}: Error finding container a43af90643342adcc950114a2e8a0ae26c7a447dd5d64932a9e789340175e09a: Status 404 returned error can't find the container with id a43af90643342adcc950114a2e8a0ae26c7a447dd5d64932a9e789340175e09a Jul 11 04:14:40.290348 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.290317 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:40.290446 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.290393 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:40.290508 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:40.290440 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:41.091331 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:41.091289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerStarted","Data":"b817b2cc2486a29159a635c492ab0d6ef65719e73553761c54082f6b756f54c4"} Jul 11 04:14:41.091778 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:41.091352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerStarted","Data":"a43af90643342adcc950114a2e8a0ae26c7a447dd5d64932a9e789340175e09a"} Jul 11 04:14:41.110497 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:41.110445 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podStartSLOduration=2.110429832 podStartE2EDuration="2.110429832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:14:41.108739963 +0000 UTC m=+1919.182356232" watchObservedRunningTime="2026-07-11 04:14:41.110429832 +0000 UTC m=+1919.184046091" Jul 11 04:14:41.135296 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:41.135259 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:41.140306 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:41.140282 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:42.096743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:42.096712 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:42.097653 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:42.097633 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:14:42.573648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:42.573618 2565 scope.go:117] "RemoveContainer" containerID="6b5e16f179411429ca145175702749280acc4348b0ee2d4b85ec7e8ec047149f" Jul 11 04:14:57.335365 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:57.334734 2565 patch_prober.go:28] interesting pod/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:53568->10.133.0.42:15021: read: connection reset by peer" start-of-body= Jul 11 04:14:57.335365 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:57.334818 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:53568->10.133.0.42:15021: read: connection reset by peer" Jul 11 04:14:58.186438 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.186403 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerID="b817b2cc2486a29159a635c492ab0d6ef65719e73553761c54082f6b756f54c4" exitCode=0 Jul 11 04:14:58.186616 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.186466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerDied","Data":"b817b2cc2486a29159a635c492ab0d6ef65719e73553761c54082f6b756f54c4"} Jul 11 04:14:58.186891 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.186876 2565 scope.go:117] "RemoveContainer" containerID="b817b2cc2486a29159a635c492ab0d6ef65719e73553761c54082f6b756f54c4" Jul 11 04:14:58.187851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.187822 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:58.187904 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.187889 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:58.187937 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:58.187918 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:14:59.193084 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:14:59.193050 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerStarted","Data":"76be02cfd653a0fdcc377270fd7bce17e57157ba44ca426ffbe02f9dcaa5b03b"} Jul 11 04:15:00.127982 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.127943 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn"] Jul 11 04:15:00.131958 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.131935 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz"] Jul 11 04:15:00.132092 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.132041 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" Jul 11 04:15:00.134130 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.134097 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:15:00.138789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.135623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" Jul 11 04:15:00.138789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.136539 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:15:00.138789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.136624 2565 patch_prober.go:28] interesting pod/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 11 04:15:00.138789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.136662 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 11 04:15:00.138789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.138738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn"] Jul 11 04:15:00.142007 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.141975 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz"] Jul 11 04:15:00.149556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.149534 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" Jul 11 04:15:00.185559 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.185530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" Jul 11 04:15:00.357708 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.357640 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn"] Jul 11 04:15:00.366432 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:15:00.366390 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9699f2_34a5_4acb_b47a_5ad8b4f79655.slice/crio-2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644 WatchSource:0}: Error finding container 2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644: Status 404 returned error can't find the container with id 2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644 Jul 11 04:15:00.393601 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:00.393578 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz"] Jul 11 04:15:00.397873 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:15:00.397850 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90d4763_7d60_43e6_a6d5_1a6b41db89a1.slice/crio-32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e WatchSource:0}: Error finding container 32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e: Status 404 returned error can't find the container with id 32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e Jul 11 04:15:01.136634 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.136591 2565 patch_prober.go:28] interesting pod/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 11 04:15:01.136863 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.136660 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 11 04:15:01.215332 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.215291 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" event={"ID":"ef9699f2-34a5-4acb-b47a-5ad8b4f79655","Type":"ContainerStarted","Data":"c6ed1f33aa53c385540e7a4d57d769b75330d81f65b569a426ec9dea5721fc52"} Jul 11 04:15:01.215515 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.215359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" event={"ID":"ef9699f2-34a5-4acb-b47a-5ad8b4f79655","Type":"ContainerStarted","Data":"2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644"} Jul 11 04:15:01.216829 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.216791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" event={"ID":"e90d4763-7d60-43e6-a6d5-1a6b41db89a1","Type":"ContainerStarted","Data":"22b7b36078e65974741c9804ee08194879a57167322a8d1dac4122ed7e461e89"} Jul 11 04:15:01.216938 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.216841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" event={"ID":"e90d4763-7d60-43e6-a6d5-1a6b41db89a1","Type":"ContainerStarted","Data":"32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e"} Jul 11 04:15:01.228930 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:01.228877 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" podStartSLOduration=1.228862356 podStartE2EDuration="1.228862356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:15:01.22856174 +0000 UTC m=+1939.302177999" watchObservedRunningTime="2026-07-11 04:15:01.228862356 +0000 UTC m=+1939.302478615" Jul 11 04:15:02.135926 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.135883 2565 patch_prober.go:28] interesting pod/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 11 04:15:02.136436 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.135983 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 11 04:15:02.223467 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.223427 2565 generic.go:356] "Generic (PLEG): container finished" podID="ef9699f2-34a5-4acb-b47a-5ad8b4f79655" containerID="c6ed1f33aa53c385540e7a4d57d769b75330d81f65b569a426ec9dea5721fc52" exitCode=0 Jul 11 04:15:02.223666 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.223522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" event={"ID":"ef9699f2-34a5-4acb-b47a-5ad8b4f79655","Type":"ContainerDied","Data":"c6ed1f33aa53c385540e7a4d57d769b75330d81f65b569a426ec9dea5721fc52"} Jul 11 04:15:02.224992 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.224966 2565 generic.go:356] "Generic (PLEG): container finished" podID="e90d4763-7d60-43e6-a6d5-1a6b41db89a1" containerID="22b7b36078e65974741c9804ee08194879a57167322a8d1dac4122ed7e461e89" exitCode=0 Jul 11 04:15:02.225096 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.225040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" event={"ID":"e90d4763-7d60-43e6-a6d5-1a6b41db89a1","Type":"ContainerDied","Data":"22b7b36078e65974741c9804ee08194879a57167322a8d1dac4122ed7e461e89"} Jul 11 04:15:02.240809 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:02.240743 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" podStartSLOduration=2.2407219449999998 podStartE2EDuration="2.240721945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:15:01.249888819 +0000 UTC m=+1939.323505079" watchObservedRunningTime="2026-07-11 04:15:02.240721945 +0000 UTC m=+1940.314338205" Jul 11 04:15:03.139405 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:03.139375 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:15:03.139827 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:03.139780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:15:03.140545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:03.140525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:15:03.384258 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:03.384232 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" Jul 11 04:15:03.413479 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:03.413453 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" Jul 11 04:15:04.236772 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.236736 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" Jul 11 04:15:04.236772 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.236751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn" event={"ID":"ef9699f2-34a5-4acb-b47a-5ad8b4f79655","Type":"ContainerDied","Data":"2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644"} Jul 11 04:15:04.237215 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.236799 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f9f6439efb44b7a16355c498a05c01a1b582c92e88df7471b2bbc00369f6644" Jul 11 04:15:04.238164 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.238139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" event={"ID":"e90d4763-7d60-43e6-a6d5-1a6b41db89a1","Type":"ContainerDied","Data":"32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e"} Jul 11 04:15:04.238266 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.238165 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29729055-skznz" Jul 11 04:15:04.238266 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:04.238186 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32849d9e3463b0cf330ec4ef6840897471fe4c13f25d3eaec09988d8edb78e8e" Jul 11 04:15:36.309912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.309826 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l"] Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310180 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90d4763-7d60-43e6-a6d5-1a6b41db89a1" containerName="cleanup" Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310192 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90d4763-7d60-43e6-a6d5-1a6b41db89a1" containerName="cleanup" Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310199 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9699f2-34a5-4acb-b47a-5ad8b4f79655" containerName="cleanup" Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310205 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9699f2-34a5-4acb-b47a-5ad8b4f79655" containerName="cleanup" Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310257 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9699f2-34a5-4acb-b47a-5ad8b4f79655" containerName="cleanup" Jul 11 04:15:36.310376 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.310264 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90d4763-7d60-43e6-a6d5-1a6b41db89a1" containerName="cleanup" Jul 11 04:15:36.318070 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.318030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:36.320286 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.320258 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:15:36.320442 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.320350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l"] Jul 11 04:15:36.403785 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.403744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l\" (UID: \"6cd28145-9761-442d-87ee-01683e92da25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:36.504704 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.504637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l\" (UID: \"6cd28145-9761-442d-87ee-01683e92da25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:36.505288 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.505265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l\" (UID: \"6cd28145-9761-442d-87ee-01683e92da25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:36.629821 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.629601 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:36.761518 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:36.761492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l"] Jul 11 04:15:36.764096 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:15:36.764069 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd28145_9761_442d_87ee_01683e92da25.slice/crio-103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666 WatchSource:0}: Error finding container 103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666: Status 404 returned error can't find the container with id 103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666 Jul 11 04:15:37.393722 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:37.393655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" event={"ID":"6cd28145-9761-442d-87ee-01683e92da25","Type":"ContainerStarted","Data":"40d04390fa46ace1d248ba57744a7ab000d97a2fd16c188fed330d37e1fe1136"} Jul 11 04:15:37.394220 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:37.393740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" event={"ID":"6cd28145-9761-442d-87ee-01683e92da25","Type":"ContainerStarted","Data":"103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666"} Jul 11 04:15:37.407022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:37.406971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" podStartSLOduration=1.406954774 podStartE2EDuration="1.406954774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:15:37.405991645 +0000 UTC m=+1975.479607903" watchObservedRunningTime="2026-07-11 04:15:37.406954774 +0000 UTC m=+1975.480571035" Jul 11 04:15:38.400032 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:38.399995 2565 generic.go:356] "Generic (PLEG): container finished" podID="6cd28145-9761-442d-87ee-01683e92da25" containerID="40d04390fa46ace1d248ba57744a7ab000d97a2fd16c188fed330d37e1fe1136" exitCode=0 Jul 11 04:15:38.400467 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:38.400085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" event={"ID":"6cd28145-9761-442d-87ee-01683e92da25","Type":"ContainerDied","Data":"40d04390fa46ace1d248ba57744a7ab000d97a2fd16c188fed330d37e1fe1136"} Jul 11 04:15:39.544411 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:39.544387 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:39.632898 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:39.632860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca\") pod \"6cd28145-9761-442d-87ee-01683e92da25\" (UID: \"6cd28145-9761-442d-87ee-01683e92da25\") " Jul 11 04:15:39.633277 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:39.633246 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cd28145-9761-442d-87ee-01683e92da25" (UID: "6cd28145-9761-442d-87ee-01683e92da25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:15:39.734377 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:39.734283 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd28145-9761-442d-87ee-01683e92da25-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:15:40.413336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:40.413297 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" Jul 11 04:15:40.413527 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:40.413294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l" event={"ID":"6cd28145-9761-442d-87ee-01683e92da25","Type":"ContainerDied","Data":"103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666"} Jul 11 04:15:40.413527 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:40.413447 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="103bab9d68cb114ef9b8ef747c85be4d45e2c789e8f53dee66818c8ebf93d666" Jul 11 04:15:42.595440 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:42.595409 2565 scope.go:117] "RemoveContainer" containerID="11fc61ce1c9ab2cc3a8dafd21f23db8344b020478ad940d43d17ee935b68c7a6" Jul 11 04:15:46.729465 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:46.729426 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn"] Jul 11 04:15:46.731950 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:46.731925 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d9f87d6a-29729055-48kpn"] Jul 11 04:15:48.426187 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:15:48.426157 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9699f2-34a5-4acb-b47a-5ad8b4f79655" path="/var/lib/kubelet/pods/ef9699f2-34a5-4acb-b47a-5ad8b4f79655/volumes" Jul 11 04:16:53.673565 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:53.673528 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:16:53.674176 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:53.673812 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" containerID="cri-o://76be02cfd653a0fdcc377270fd7bce17e57157ba44ca426ffbe02f9dcaa5b03b" gracePeriod=30 Jul 11 04:16:58.798416 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.798385 2565 generic.go:356] "Generic (PLEG): container finished" podID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerID="76be02cfd653a0fdcc377270fd7bce17e57157ba44ca426ffbe02f9dcaa5b03b" exitCode=0 Jul 11 04:16:58.798851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.798435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerDied","Data":"76be02cfd653a0fdcc377270fd7bce17e57157ba44ca426ffbe02f9dcaa5b03b"} Jul 11 04:16:58.798851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.798487 2565 scope.go:117] "RemoveContainer" containerID="b817b2cc2486a29159a635c492ab0d6ef65719e73553761c54082f6b756f54c4" Jul 11 04:16:58.931294 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.931268 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:16:58.956667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956600 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.956667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956627 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.956667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956651 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grg8g\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.956667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956666 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956800 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956863 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956926 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956930 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956940 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:16:58.957001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.956964 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert\") pod \"f7fbc239-afcb-48d9-aa89-78f602e5c949\" (UID: \"f7fbc239-afcb-48d9-aa89-78f602e5c949\") " Jul 11 04:16:58.957374 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957121 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:16:58.957434 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957370 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:58.957434 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957388 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:58.957434 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957402 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:58.957571 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957458 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:16:58.957743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.957718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:16:58.959401 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.959368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g" (OuterVolumeSpecName: "kube-api-access-grg8g") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "kube-api-access-grg8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:16:58.959518 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.959430 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:16:58.959880 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.959851 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:16:58.959977 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:58.959948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7fbc239-afcb-48d9-aa89-78f602e5c949" (UID: "f7fbc239-afcb-48d9-aa89-78f602e5c949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:16:59.058651 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058618 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grg8g\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-kube-api-access-grg8g\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.058651 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058644 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.058651 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058653 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.058651 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058661 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.058928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058669 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7fbc239-afcb-48d9-aa89-78f602e5c949-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.058928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.058706 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7fbc239-afcb-48d9-aa89-78f602e5c949-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:16:59.804841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.804803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" event={"ID":"f7fbc239-afcb-48d9-aa89-78f602e5c949","Type":"ContainerDied","Data":"a43af90643342adcc950114a2e8a0ae26c7a447dd5d64932a9e789340175e09a"} Jul 11 04:16:59.805330 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.804869 2565 scope.go:117] "RemoveContainer" containerID="76be02cfd653a0fdcc377270fd7bce17e57157ba44ca426ffbe02f9dcaa5b03b" Jul 11 04:16:59.805330 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.804821 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v" Jul 11 04:16:59.826046 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.826005 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:16:59.832113 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:16:59.832090 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d9f87d6a-openshift-default-66dc8747d7-tkp6v"] Jul 11 04:17:00.426128 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:00.426093 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" path="/var/lib/kubelet/pods/f7fbc239-afcb-48d9-aa89-78f602e5c949/volumes" Jul 11 04:17:42.537244 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:42.537159 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:17:42.542303 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:42.542280 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:17:42.622034 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:42.621996 2565 scope.go:117] "RemoveContainer" containerID="6403256c1791bacb09a7add3a3a861f6923823010a13edada21902305ae98c86" Jul 11 04:17:47.898089 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898054 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898388 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd28145-9761-442d-87ee-01683e92da25" containerName="revoke-keys" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898399 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd28145-9761-442d-87ee-01683e92da25" containerName="revoke-keys" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898414 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898419 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898427 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898432 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898482 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd28145-9761-442d-87ee-01683e92da25" containerName="revoke-keys" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898489 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.898544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.898497 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fbc239-afcb-48d9-aa89-78f602e5c949" containerName="istio-proxy" Jul 11 04:17:47.901528 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.901508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.903775 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.903755 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-888089-openshift-default-dockercfg-xjctd\"" Jul 11 04:17:47.911270 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.911246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:17:47.995926 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.995886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.995926 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.995926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.995946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txsb\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:47.996144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:47.996136 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.096857 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.096857 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txsb\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.096988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097124 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097704 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.097755 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.097723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.100002 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.099980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.100139 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.100121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.104983 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.104956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.105105 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.105087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txsb\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb\") pod \"e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.214252 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.214150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:48.352036 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.352007 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:17:48.355087 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:17:48.355062 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40bcaef_35a3_40ab_a12b_d0d04fb493f4.slice/crio-4fd349242773e2f5991379d84294ce35299ac9ef041d18e05411239212c05b24 WatchSource:0}: Error finding container 4fd349242773e2f5991379d84294ce35299ac9ef041d18e05411239212c05b24: Status 404 returned error can't find the container with id 4fd349242773e2f5991379d84294ce35299ac9ef041d18e05411239212c05b24 Jul 11 04:17:48.357163 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.357130 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:48.357250 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.357212 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:48.357250 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:48.357242 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:49.048203 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:49.048171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerStarted","Data":"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32"} Jul 11 04:17:49.048578 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:49.048228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerStarted","Data":"4fd349242773e2f5991379d84294ce35299ac9ef041d18e05411239212c05b24"} Jul 11 04:17:49.066598 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:49.066546 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podStartSLOduration=2.066528615 podStartE2EDuration="2.066528615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:17:49.065516508 +0000 UTC m=+2107.139132780" watchObservedRunningTime="2026-07-11 04:17:49.066528615 +0000 UTC m=+2107.140144875" Jul 11 04:17:49.215216 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:49.215189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:49.220177 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:49.220152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:50.053435 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:50.053356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:50.054348 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:50.054332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:17:56.164501 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.164468 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:17:56.168378 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.168357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.170853 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.170821 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-075325-openshift-default-dockercfg-8b7cv\"" Jul 11 04:17:56.180439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.180409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:17:56.269570 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.269790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7lv\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.270116 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.270116 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.269836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7lv\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371726 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371726 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371726 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371726 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371671 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371937 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.371996 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.371977 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.372155 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.372048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.373671 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.373650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.373909 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.373891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.378990 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.378965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.378990 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.378980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7lv\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv\") pod \"e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.482316 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.482227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:56.625060 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.625033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:17:56.627327 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:17:56.627299 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dac9464_fa6c_4636_b3f7_54365fbd0056.slice/crio-74ae27602d2039654b80174cf279cac2865eb6fee6e05d1ae50918f173416a57 WatchSource:0}: Error finding container 74ae27602d2039654b80174cf279cac2865eb6fee6e05d1ae50918f173416a57: Status 404 returned error can't find the container with id 74ae27602d2039654b80174cf279cac2865eb6fee6e05d1ae50918f173416a57 Jul 11 04:17:56.629547 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.629509 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:56.629642 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.629611 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:56.629738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:56.629650 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:17:57.088355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:57.088322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerStarted","Data":"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62"} Jul 11 04:17:57.088534 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:57.088374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerStarted","Data":"74ae27602d2039654b80174cf279cac2865eb6fee6e05d1ae50918f173416a57"} Jul 11 04:17:57.107858 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:57.107805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podStartSLOduration=1.107784032 podStartE2EDuration="1.107784032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:17:57.105591419 +0000 UTC m=+2115.179207679" watchObservedRunningTime="2026-07-11 04:17:57.107784032 +0000 UTC m=+2115.181400295" Jul 11 04:17:57.482626 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:57.482585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:57.487931 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:57.487904 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:58.092987 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:58.092959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:17:58.093847 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:17:58.093828 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:18:06.095881 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:06.095839 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 11 04:18:06.096308 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:06.095918 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 11 04:18:08.155858 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.155823 2565 generic.go:356] "Generic (PLEG): container finished" podID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerID="03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32" exitCode=0 Jul 11 04:18:08.156484 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.156460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerDied","Data":"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32"} Jul 11 04:18:08.157144 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.157124 2565 scope.go:117] "RemoveContainer" containerID="03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32" Jul 11 04:18:08.159235 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.159180 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:08.159349 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.159289 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:08.159349 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:08.159338 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:09.166590 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:09.166532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerStarted","Data":"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09"} Jul 11 04:18:09.215100 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:09.215059 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:18:09.215301 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:09.215279 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 11 04:18:09.215355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:09.215326 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 11 04:18:10.214641 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:10.214509 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 11 04:18:10.214641 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:10.214564 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 11 04:18:11.215439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:11.215395 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 11 04:18:11.215907 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:11.215467 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 11 04:18:12.215546 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:12.215511 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 11 04:18:12.215984 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:12.215571 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 11 04:18:13.219082 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:13.219039 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:18:13.219622 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:13.219293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:18:13.219981 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:13.219960 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:18:14.134876 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:14.134832 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 11 04:18:14.135205 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:14.134897 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 11 04:18:15.202925 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.202895 2565 generic.go:356] "Generic (PLEG): container finished" podID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerID="9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62" exitCode=0 Jul 11 04:18:15.203334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.202970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerDied","Data":"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62"} Jul 11 04:18:15.203423 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.203407 2565 scope.go:117] "RemoveContainer" containerID="9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62" Jul 11 04:18:15.204446 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.204421 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:15.204516 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.204485 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:15.204557 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:15.204519 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:18:16.209850 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:16.209817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerStarted","Data":"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60"} Jul 11 04:18:16.483104 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:16.483000 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:18:16.483289 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:16.483266 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 11 04:18:16.483349 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:16.483315 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 11 04:18:17.483179 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:17.483136 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 11 04:18:17.483578 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:17.483224 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 11 04:18:18.482908 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:18.482871 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 11 04:18:18.483098 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:18.482944 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 11 04:18:19.483457 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:19.483413 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 11 04:18:19.483895 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:19.483490 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 11 04:18:20.486602 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:20.486573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:18:20.487063 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:20.486994 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:18:20.487708 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:18:20.487671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:19:13.471544 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.471508 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck"] Jul 11 04:19:13.475242 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.475223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:13.477447 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.477430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:19:13.481928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.481899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck"] Jul 11 04:19:13.541370 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.541331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-888089-gw9ck\" (UID: \"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:13.642743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.642709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-888089-gw9ck\" (UID: \"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:13.643362 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.643344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-888089-gw9ck\" (UID: \"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:13.788480 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.788392 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:13.926345 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:13.926315 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck"] Jul 11 04:19:13.928228 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:19:13.928201 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5f9c65_a7c1_4b21_9149_c3d51a722ebf.slice/crio-dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89 WatchSource:0}: Error finding container dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89: Status 404 returned error can't find the container with id dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89 Jul 11 04:19:14.489024 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:14.488921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" event={"ID":"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf","Type":"ContainerStarted","Data":"c9e5999852d1cb547be6c733a4187f3345542031ba65f9caede27060abfd89ab"} Jul 11 04:19:14.489024 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:14.488980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" event={"ID":"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf","Type":"ContainerStarted","Data":"dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89"} Jul 11 04:19:14.504360 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:14.504312 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" podStartSLOduration=1.504294713 podStartE2EDuration="1.504294713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:19:14.502829486 +0000 UTC m=+2192.576445747" watchObservedRunningTime="2026-07-11 04:19:14.504294713 +0000 UTC m=+2192.577910972" Jul 11 04:19:15.496358 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:15.496322 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f5f9c65-a7c1-4b21-9149-c3d51a722ebf" containerID="c9e5999852d1cb547be6c733a4187f3345542031ba65f9caede27060abfd89ab" exitCode=0 Jul 11 04:19:15.496755 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:15.496376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" event={"ID":"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf","Type":"ContainerDied","Data":"c9e5999852d1cb547be6c733a4187f3345542031ba65f9caede27060abfd89ab"} Jul 11 04:19:16.659906 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:16.659873 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:16.773233 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:16.773142 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca\") pod \"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf\" (UID: \"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf\") " Jul 11 04:19:16.773500 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:16.773477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f5f9c65-a7c1-4b21-9149-c3d51a722ebf" (UID: "1f5f9c65-a7c1-4b21-9149-c3d51a722ebf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:19:16.874195 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:16.874159 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5f9c65-a7c1-4b21-9149-c3d51a722ebf-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:17.508521 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:17.508483 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" Jul 11 04:19:17.508521 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:17.508496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-888089-gw9ck" event={"ID":"1f5f9c65-a7c1-4b21-9149-c3d51a722ebf","Type":"ContainerDied","Data":"dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89"} Jul 11 04:19:17.508769 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:17.508544 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc96d7aa5352f8bede0acb4bc68ed05385a83f38e5be1db44f9a3048380b3b89" Jul 11 04:19:39.223192 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:39.223155 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:19:39.223738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:39.223414 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" containerID="cri-o://42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09" gracePeriod=30 Jul 11 04:19:43.221447 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:43.221402 2565 patch_prober.go:28] interesting pod/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:19:43.221851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:43.221478 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:19:44.479952 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.479928 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:19:44.644101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644068 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9txsb\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644140 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644167 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644209 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644248 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644373 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644366 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token\") pod \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\" (UID: \"c40bcaef-35a3-40ab-a12b-d0d04fb493f4\") " Jul 11 04:19:44.644743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644615 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:19:44.644743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:19:44.644743 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644596 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:19:44.644917 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.644762 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.645169 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645139 2565 generic.go:356] "Generic (PLEG): container finished" podID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerID="42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09" exitCode=0 Jul 11 04:19:44.645288 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerDied","Data":"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09"} Jul 11 04:19:44.645288 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645266 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:19:44.645418 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" event={"ID":"c40bcaef-35a3-40ab-a12b-d0d04fb493f4","Type":"ContainerDied","Data":"4fd349242773e2f5991379d84294ce35299ac9ef041d18e05411239212c05b24"} Jul 11 04:19:44.645418 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645327 2565 scope.go:117] "RemoveContainer" containerID="42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09" Jul 11 04:19:44.645418 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645363 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk" Jul 11 04:19:44.645916 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.645890 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:19:44.646921 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.646898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb" (OuterVolumeSpecName: "kube-api-access-9txsb") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "kube-api-access-9txsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:19:44.647203 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.647181 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:19:44.647298 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.647197 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:19:44.647298 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.647246 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c40bcaef-35a3-40ab-a12b-d0d04fb493f4" (UID: "c40bcaef-35a3-40ab-a12b-d0d04fb493f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:19:44.666088 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.666056 2565 scope.go:117] "RemoveContainer" containerID="03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32" Jul 11 04:19:44.674964 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.674948 2565 scope.go:117] "RemoveContainer" containerID="42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09" Jul 11 04:19:44.675244 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:19:44.675221 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09\": container with ID starting with 42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09 not found: ID does not exist" containerID="42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09" Jul 11 04:19:44.675282 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.675256 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09"} err="failed to get container status \"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09\": rpc error: code = NotFound desc = could not find container \"42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09\": container with ID starting with 42d76fc02b325e7a82a4ee311f363741b190ec59ec28eb2060b0c66bb10afb09 not found: ID does not exist" Jul 11 04:19:44.675282 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.675274 2565 scope.go:117] "RemoveContainer" containerID="03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32" Jul 11 04:19:44.675522 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:19:44.675504 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32\": container with ID starting with 03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32 not found: ID does not exist" containerID="03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32" Jul 11 04:19:44.675559 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.675528 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32"} err="failed to get container status \"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32\": rpc error: code = NotFound desc = could not find container \"03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32\": container with ID starting with 03aa76eaa20955c7e0870799c6911ffb590a97c45be74383b35d7a16041cce32 not found: ID does not exist" Jul 11 04:19:44.745264 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745227 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745264 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745258 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txsb\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-kube-api-access-9txsb\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745264 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745268 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745279 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745288 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745296 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745304 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.745482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.745311 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bcaef-35a3-40ab-a12b-d0d04fb493f4-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:19:44.966877 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.966832 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:19:44.969960 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:44.969935 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-888089-openshift-default-68b44bdb4c-t6dpk"] Jul 11 04:19:46.426186 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:19:46.426151 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" path="/var/lib/kubelet/pods/c40bcaef-35a3-40ab-a12b-d0d04fb493f4/volumes" Jul 11 04:20:05.390051 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:05.389953 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:20:05.390517 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:05.390211 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" containerID="cri-o://2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60" gracePeriod=30 Jul 11 04:20:05.490614 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:05.490573 2565 patch_prober.go:28] interesting pod/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:20:05.490834 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:05.490647 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:20:06.318194 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318157 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:20:06.318658 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318642 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:06.318773 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318660 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:06.318773 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318714 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5f9c65-a7c1-4b21-9149-c3d51a722ebf" containerName="revoke-keys" Jul 11 04:20:06.318773 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318722 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5f9c65-a7c1-4b21-9149-c3d51a722ebf" containerName="revoke-keys" Jul 11 04:20:06.318928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318794 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:06.318928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318809 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:06.318928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.318820 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5f9c65-a7c1-4b21-9149-c3d51a722ebf" containerName="revoke-keys" Jul 11 04:20:06.322433 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.322412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.324799 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.324779 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-20f328-openshift-default-dockercfg-nxbbr\"" Jul 11 04:20:06.331816 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.331784 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:20:06.442414 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442414 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.442912 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.442714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmn7\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543540 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmn7\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.543746 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544016 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544016 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544016 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.543840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544148 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.544123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544187 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.544137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544239 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.544181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544357 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.544334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.544697 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.544627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.546286 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.546265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.546577 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.546559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.550777 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.550749 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.551135 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.551116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmn7\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7\") pod \"e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.633668 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.633633 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:06.782551 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.782524 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:20:06.788809 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.788769 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:06.788924 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.788857 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:06.788924 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:06.788906 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:07.762772 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:07.762734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerStarted","Data":"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127"} Jul 11 04:20:07.763171 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:07.762821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerStarted","Data":"8a298d4624e914ab9ec9ca8411103743dbac77eb2f1f1e57baa81e468d4e3c9e"} Jul 11 04:20:07.783844 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:07.783792 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podStartSLOduration=1.783774067 podStartE2EDuration="1.783774067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:20:07.781567653 +0000 UTC m=+2245.855183922" watchObservedRunningTime="2026-07-11 04:20:07.783774067 +0000 UTC m=+2245.857390327" Jul 11 04:20:08.634060 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:08.634023 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:08.639361 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:08.639332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:08.769253 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:08.769222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:08.770337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:08.770318 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:10.658996 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.658912 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:20:10.780560 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.780524 2565 generic.go:356] "Generic (PLEG): container finished" podID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerID="2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60" exitCode=0 Jul 11 04:20:10.780768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.780570 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" Jul 11 04:20:10.780768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.780600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerDied","Data":"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60"} Jul 11 04:20:10.780768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.780649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv" event={"ID":"3dac9464-fa6c-4636-b3f7-54365fbd0056","Type":"ContainerDied","Data":"74ae27602d2039654b80174cf279cac2865eb6fee6e05d1ae50918f173416a57"} Jul 11 04:20:10.780768 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.780688 2565 scope.go:117] "RemoveContainer" containerID="2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60" Jul 11 04:20:10.787922 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.787897 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.787944 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.787973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788020 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt7lv\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788044 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788061 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788337 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788222 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket\") pod \"3dac9464-fa6c-4636-b3f7-54365fbd0056\" (UID: \"3dac9464-fa6c-4636-b3f7-54365fbd0056\") " Jul 11 04:20:10.788649 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788623 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:20:10.788941 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788902 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:20:10.788941 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788913 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:20:10.789074 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.788959 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data" (OuterVolumeSpecName: "istio-data") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:20:10.789143 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.789118 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:20:10.790549 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.790463 2565 scope.go:117] "RemoveContainer" containerID="9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62" Jul 11 04:20:10.791086 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.791061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token" (OuterVolumeSpecName: "istio-token") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:20:10.791299 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.791275 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:20:10.791619 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.791557 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv" (OuterVolumeSpecName: "kube-api-access-rt7lv") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "kube-api-access-rt7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:20:10.791619 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.791559 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3dac9464-fa6c-4636-b3f7-54365fbd0056" (UID: "3dac9464-fa6c-4636-b3f7-54365fbd0056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:20:10.809168 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.809142 2565 scope.go:117] "RemoveContainer" containerID="2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60" Jul 11 04:20:10.809481 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:20:10.809461 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60\": container with ID starting with 2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60 not found: ID does not exist" containerID="2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60" Jul 11 04:20:10.809549 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.809491 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60"} err="failed to get container status \"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60\": rpc error: code = NotFound desc = could not find container \"2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60\": container with ID starting with 2bc0b1fc1de58395b2d7c3c54ccb3eb8ce09b1631952ce3b722f4e08752a8f60 not found: ID does not exist" Jul 11 04:20:10.809549 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.809509 2565 scope.go:117] "RemoveContainer" containerID="9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62" Jul 11 04:20:10.809765 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:20:10.809744 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62\": container with ID starting with 9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62 not found: ID does not exist" containerID="9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62" Jul 11 04:20:10.809818 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.809772 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62"} err="failed to get container status \"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62\": rpc error: code = NotFound desc = could not find container \"9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62\": container with ID starting with 9649f0503bcb187cf74baa5d414c373e4bf990d400a5e4a857adbb1482b3cb62 not found: ID does not exist" Jul 11 04:20:10.889841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889797 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.889841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889835 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889852 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889868 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dac9464-fa6c-4636-b3f7-54365fbd0056-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889881 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889892 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7lv\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-kube-api-access-rt7lv\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889905 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889918 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:10.890101 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:10.889930 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dac9464-fa6c-4636-b3f7-54365fbd0056-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:11.102667 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:11.102631 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:20:11.106200 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:11.106172 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-075325-openshift-default-687c5985bc-7k7rv"] Jul 11 04:20:12.425778 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:12.425743 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" path="/var/lib/kubelet/pods/3dac9464-fa6c-4636-b3f7-54365fbd0056/volumes" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957160 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957745 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957766 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957794 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957803 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957818 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957827 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40bcaef-35a3-40ab-a12b-d0d04fb493f4" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957940 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.959334 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.957953 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dac9464-fa6c-4636-b3f7-54365fbd0056" containerName="istio-proxy" Jul 11 04:20:14.973716 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.966001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:14.973716 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.972603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e88abb-openshift-default-dockercfg-dlqqb\"" Jul 11 04:20:14.974094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:14.974061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:20:15.027783 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.027747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.027939 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.027797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.027939 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.027827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.027939 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.027913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.028050 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.027953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.028050 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.028015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.028050 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.028042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.028145 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.028085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pxz\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.028145 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.028126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129098 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129638 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pxz\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129638 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129638 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129850 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129850 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.129955 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.129913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.131851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.131820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.132197 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.132179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.144310 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.143945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pxz\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.144472 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.144349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token\") pod \"e2e-api-b-e88abb-openshift-default-db5589557-ktmp6\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.285012 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.284915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:15.451738 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.450426 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:20:15.463791 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:20:15.463758 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cacb612_c68b_4e20_b0ac_cf94a5435b0a.slice/crio-dcff2f83d2d2d6513ab5dc46bd0687001acfba7e2600facdf26c99e03944f4ba WatchSource:0}: Error finding container dcff2f83d2d2d6513ab5dc46bd0687001acfba7e2600facdf26c99e03944f4ba: Status 404 returned error can't find the container with id dcff2f83d2d2d6513ab5dc46bd0687001acfba7e2600facdf26c99e03944f4ba Jul 11 04:20:15.465942 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.465908 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:15.466044 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.465982 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:15.466044 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.466017 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:15.807110 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.807071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerStarted","Data":"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b"} Jul 11 04:20:15.807286 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.807134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerStarted","Data":"dcff2f83d2d2d6513ab5dc46bd0687001acfba7e2600facdf26c99e03944f4ba"} Jul 11 04:20:15.828091 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:15.828031 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podStartSLOduration=1.828014868 podStartE2EDuration="1.828014868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:20:15.824617929 +0000 UTC m=+2253.898234226" watchObservedRunningTime="2026-07-11 04:20:15.828014868 +0000 UTC m=+2253.901631127" Jul 11 04:20:16.285960 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:16.285916 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:16.291043 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:16.291017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:16.812170 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:16.812134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:16.813273 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:16.813255 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:22.846660 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.846624 2565 generic.go:356] "Generic (PLEG): container finished" podID="79544898-0382-41bc-8239-4449f5262a4e" containerID="991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127" exitCode=0 Jul 11 04:20:22.847120 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.846709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerDied","Data":"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127"} Jul 11 04:20:22.847202 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.847187 2565 scope.go:117] "RemoveContainer" containerID="991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127" Jul 11 04:20:22.848175 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.848147 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:22.848233 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.848213 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:22.848279 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:22.848244 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:23.770117 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:23.770079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:23.853911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:23.853873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerStarted","Data":"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c"} Jul 11 04:20:24.635072 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:24.634986 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:24.636475 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:24.636144 2565 patch_prober.go:28] interesting pod/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 11 04:20:24.636475 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:24.636200 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 11 04:20:25.634282 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:25.634245 2565 patch_prober.go:28] interesting pod/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 11 04:20:25.635266 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:25.634846 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 11 04:20:26.634163 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:26.634120 2565 patch_prober.go:28] interesting pod/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 11 04:20:26.634395 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:26.634198 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 11 04:20:27.635323 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:27.634860 2565 patch_prober.go:28] interesting pod/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 11 04:20:27.635323 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:27.634926 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 11 04:20:28.659561 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:28.659520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:28.660438 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:28.660408 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:28.661484 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:28.661461 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:20:30.389938 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.389901 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d"] Jul 11 04:20:30.394066 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.394043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:30.396414 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.396393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:20:30.400554 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.400521 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d"] Jul 11 04:20:30.493907 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.493866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-20f328-tpp8d\" (UID: \"304ae2f0-2622-4ffc-a201-5f008eecfbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:30.595243 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.595194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-20f328-tpp8d\" (UID: \"304ae2f0-2622-4ffc-a201-5f008eecfbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:30.595928 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.595908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-20f328-tpp8d\" (UID: \"304ae2f0-2622-4ffc-a201-5f008eecfbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:30.705519 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.705430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:30.892308 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:30.892278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d"] Jul 11 04:20:30.894887 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:20:30.894855 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304ae2f0_2622_4ffc_a201_5f008eecfbae.slice/crio-6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be WatchSource:0}: Error finding container 6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be: Status 404 returned error can't find the container with id 6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be Jul 11 04:20:31.900890 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:31.900854 2565 generic.go:356] "Generic (PLEG): container finished" podID="304ae2f0-2622-4ffc-a201-5f008eecfbae" containerID="77b031dd33f36cfb63b5ae59f7b22428db0c4ad690911d06cdc85048cc96105a" exitCode=0 Jul 11 04:20:31.901327 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:31.900944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" event={"ID":"304ae2f0-2622-4ffc-a201-5f008eecfbae","Type":"ContainerDied","Data":"77b031dd33f36cfb63b5ae59f7b22428db0c4ad690911d06cdc85048cc96105a"} Jul 11 04:20:31.901327 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:31.901016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" event={"ID":"304ae2f0-2622-4ffc-a201-5f008eecfbae","Type":"ContainerStarted","Data":"6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be"} Jul 11 04:20:32.853908 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:32.853851 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 11 04:20:32.854133 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:32.853930 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 11 04:20:33.053770 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.053743 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:33.118299 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.118212 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca\") pod \"304ae2f0-2622-4ffc-a201-5f008eecfbae\" (UID: \"304ae2f0-2622-4ffc-a201-5f008eecfbae\") " Jul 11 04:20:33.118639 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.118618 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "304ae2f0-2622-4ffc-a201-5f008eecfbae" (UID: "304ae2f0-2622-4ffc-a201-5f008eecfbae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:20:33.219299 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.219256 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/304ae2f0-2622-4ffc-a201-5f008eecfbae-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:20:33.913616 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.913583 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" Jul 11 04:20:33.913798 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.913590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-20f328-tpp8d" event={"ID":"304ae2f0-2622-4ffc-a201-5f008eecfbae","Type":"ContainerDied","Data":"6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be"} Jul 11 04:20:33.913798 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:33.913729 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6947b719a4d2cd1bb6fbc868352fc60be20020eda7aaa77356ee8fd2f367e4be" Jul 11 04:20:34.919962 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.919930 2565 generic.go:356] "Generic (PLEG): container finished" podID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerID="d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b" exitCode=0 Jul 11 04:20:34.920401 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.920011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerDied","Data":"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b"} Jul 11 04:20:34.920445 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.920410 2565 scope.go:117] "RemoveContainer" containerID="d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b" Jul 11 04:20:34.921405 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.921376 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:34.921482 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.921464 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:34.921518 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:34.921506 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:20:35.927234 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:35.927201 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerStarted","Data":"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a"} Jul 11 04:20:36.285328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:36.285229 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:36.285543 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:36.285521 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 11 04:20:36.285592 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:36.285568 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 11 04:20:37.285510 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:37.285469 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 11 04:20:37.285957 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:37.285535 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 11 04:20:38.285704 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:38.285641 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 11 04:20:38.286118 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:38.285730 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 11 04:20:39.285823 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:39.285786 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 11 04:20:39.286241 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:39.285846 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 11 04:20:40.289819 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:40.289784 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:40.290228 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:40.290178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:40.290841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:40.290820 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:20:56.148419 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:56.148380 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:20:56.148941 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:56.148666 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" containerID="cri-o://f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c" gracePeriod=30 Jul 11 04:20:58.662622 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:58.662588 2565 patch_prober.go:28] interesting pod/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 11 04:20:58.663000 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:20:58.662659 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:21:01.404040 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.404017 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:21:01.493710 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493595 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493710 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493654 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493710 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493709 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493728 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493744 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493853 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvmn7\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.493995 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493896 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo\") pod \"79544898-0382-41bc-8239-4449f5262a4e\" (UID: \"79544898-0382-41bc-8239-4449f5262a4e\") " Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.493989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494054 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494212 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494270 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.494328 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494289 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.494818 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.494627 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:01.496033 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.496005 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:01.496157 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.496097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7" (OuterVolumeSpecName: "kube-api-access-nvmn7") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "kube-api-access-nvmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:21:01.496649 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.496628 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:21:01.496649 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.496636 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "79544898-0382-41bc-8239-4449f5262a4e" (UID: "79544898-0382-41bc-8239-4449f5262a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:21:01.595439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595402 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595428 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79544898-0382-41bc-8239-4449f5262a4e-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595438 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595439 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595446 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmn7\" (UniqueName: \"kubernetes.io/projected/79544898-0382-41bc-8239-4449f5262a4e-kube-api-access-nvmn7\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595826 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595455 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79544898-0382-41bc-8239-4449f5262a4e-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595826 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595463 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:01.595826 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:01.595471 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79544898-0382-41bc-8239-4449f5262a4e-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:02.066814 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.066779 2565 generic.go:356] "Generic (PLEG): container finished" podID="79544898-0382-41bc-8239-4449f5262a4e" containerID="f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c" exitCode=0 Jul 11 04:21:02.067020 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.066841 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" Jul 11 04:21:02.067020 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.066852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerDied","Data":"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c"} Jul 11 04:21:02.067020 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.066895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6" event={"ID":"79544898-0382-41bc-8239-4449f5262a4e","Type":"ContainerDied","Data":"8a298d4624e914ab9ec9ca8411103743dbac77eb2f1f1e57baa81e468d4e3c9e"} Jul 11 04:21:02.067020 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.066918 2565 scope.go:117] "RemoveContainer" containerID="f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c" Jul 11 04:21:02.076520 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.076501 2565 scope.go:117] "RemoveContainer" containerID="991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127" Jul 11 04:21:02.084613 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.084593 2565 scope.go:117] "RemoveContainer" containerID="f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c" Jul 11 04:21:02.084900 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:21:02.084880 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c\": container with ID starting with f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c not found: ID does not exist" containerID="f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c" Jul 11 04:21:02.084976 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.084909 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c"} err="failed to get container status \"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c\": rpc error: code = NotFound desc = could not find container \"f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c\": container with ID starting with f6b0bac327c074715ad5346843dafdaa918063142c9916cecc4e97fcc1657f2c not found: ID does not exist" Jul 11 04:21:02.084976 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.084926 2565 scope.go:117] "RemoveContainer" containerID="991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127" Jul 11 04:21:02.085181 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:21:02.085165 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127\": container with ID starting with 991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127 not found: ID does not exist" containerID="991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127" Jul 11 04:21:02.085220 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.085188 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127"} err="failed to get container status \"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127\": rpc error: code = NotFound desc = could not find container \"991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127\": container with ID starting with 991cdbedde9ed19ee2a486a6e6eecc4eaa060a8fa8c55cb6a06e4540cbc04127 not found: ID does not exist" Jul 11 04:21:02.089160 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.089137 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:21:02.093649 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.093628 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-20f328-openshift-default-6fcc8dbc4f-pqnv6"] Jul 11 04:21:02.433432 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:02.433395 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79544898-0382-41bc-8239-4449f5262a4e" path="/var/lib/kubelet/pods/79544898-0382-41bc-8239-4449f5262a4e/volumes" Jul 11 04:21:22.301874 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:22.301840 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:21:22.302455 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:22.302072 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" containerID="cri-o://c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a" gracePeriod=30 Jul 11 04:21:23.223093 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223057 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:21:23.223450 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223434 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:23.223513 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223453 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:23.223513 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223467 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304ae2f0-2622-4ffc-a201-5f008eecfbae" containerName="revoke-keys" Jul 11 04:21:23.223513 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223473 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="304ae2f0-2622-4ffc-a201-5f008eecfbae" containerName="revoke-keys" Jul 11 04:21:23.223663 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223538 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:23.223663 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223547 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:23.223663 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.223555 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="304ae2f0-2622-4ffc-a201-5f008eecfbae" containerName="revoke-keys" Jul 11 04:21:23.227163 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.227133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.233059 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.233031 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6ee78d-openshift-default-dockercfg-j5s5k\"" Jul 11 04:21:23.237068 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.237041 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:21:23.296266 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296266 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296538 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296293 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gc75\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296538 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296538 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296538 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296763 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296763 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.296763 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.296607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.397720 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.397656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.397735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.397761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.397837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.397871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gc75\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398671 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398671 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398671 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398443 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398671 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.398877 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.398722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.400348 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.400325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.400455 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.400418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.406160 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.406133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.406270 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.406226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gc75\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75\") pod \"e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.544479 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.544386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:23.708333 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.708301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:21:23.709754 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:21:23.709723 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc33284_0ded_4ad4_9e38_8ff45efabf12.slice/crio-dc8df2fec59f933f82e90b587e7bbceb2141ae32fe3e942782bc99e726c57f17 WatchSource:0}: Error finding container dc8df2fec59f933f82e90b587e7bbceb2141ae32fe3e942782bc99e726c57f17: Status 404 returned error can't find the container with id dc8df2fec59f933f82e90b587e7bbceb2141ae32fe3e942782bc99e726c57f17 Jul 11 04:21:23.711920 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.711879 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:23.712001 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.711974 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:23.712038 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:23.712019 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:24.179431 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:24.179397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerStarted","Data":"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458"} Jul 11 04:21:24.179634 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:24.179453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerStarted","Data":"dc8df2fec59f933f82e90b587e7bbceb2141ae32fe3e942782bc99e726c57f17"} Jul 11 04:21:24.199022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:24.198967 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podStartSLOduration=1.198950123 podStartE2EDuration="1.198950123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:21:24.198511321 +0000 UTC m=+2322.272127592" watchObservedRunningTime="2026-07-11 04:21:24.198950123 +0000 UTC m=+2322.272566381" Jul 11 04:21:24.545195 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:24.545099 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:24.550609 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:24.550583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:25.185242 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:25.185212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:25.186265 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:25.186248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:25.292774 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:25.292736 2565 patch_prober.go:28] interesting pod/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 04:21:25.292939 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:25.292791 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 04:21:27.554970 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.554945 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:21:27.641721 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641654 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.641911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.641911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.641911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641831 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.641911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641864 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52pxz\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.641911 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.642204 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641943 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.642204 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.641980 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:27.642204 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642022 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.642204 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642053 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token\") pod \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\" (UID: \"3cacb612-c68b-4e20-b0ac-cf94a5435b0a\") " Jul 11 04:21:27.642204 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642109 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:27.642462 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642346 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.642462 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642365 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.642462 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642434 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:21:27.642614 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642457 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:27.642776 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.642755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:27.644188 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.644167 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:21:27.644318 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.644288 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:21:27.644641 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.644614 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz" (OuterVolumeSpecName: "kube-api-access-52pxz") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "kube-api-access-52pxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:21:27.644741 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.644638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cacb612-c68b-4e20-b0ac-cf94a5435b0a" (UID: "3cacb612-c68b-4e20-b0ac-cf94a5435b0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:21:27.743477 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743437 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743477 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743470 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743477 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743480 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52pxz\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-kube-api-access-52pxz\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743477 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743489 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743804 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743499 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743804 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743506 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:27.743804 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:27.743514 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cacb612-c68b-4e20-b0ac-cf94a5435b0a-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:21:28.200268 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.200233 2565 generic.go:356] "Generic (PLEG): container finished" podID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerID="c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a" exitCode=0 Jul 11 04:21:28.200453 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.200277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerDied","Data":"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a"} Jul 11 04:21:28.200453 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.200320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" event={"ID":"3cacb612-c68b-4e20-b0ac-cf94a5435b0a","Type":"ContainerDied","Data":"dcff2f83d2d2d6513ab5dc46bd0687001acfba7e2600facdf26c99e03944f4ba"} Jul 11 04:21:28.200453 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.200346 2565 scope.go:117] "RemoveContainer" containerID="c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a" Jul 11 04:21:28.200453 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.200279 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6" Jul 11 04:21:28.214372 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.214342 2565 scope.go:117] "RemoveContainer" containerID="d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b" Jul 11 04:21:28.223281 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.223251 2565 scope.go:117] "RemoveContainer" containerID="c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a" Jul 11 04:21:28.223594 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:21:28.223570 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a\": container with ID starting with c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a not found: ID does not exist" containerID="c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a" Jul 11 04:21:28.223697 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.223603 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a"} err="failed to get container status \"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a\": rpc error: code = NotFound desc = could not find container \"c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a\": container with ID starting with c51d3e5de000934c760bb72257aa36a73008e664a41c0cf5e72f30f4e1f7808a not found: ID does not exist" Jul 11 04:21:28.223697 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.223622 2565 scope.go:117] "RemoveContainer" containerID="d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b" Jul 11 04:21:28.223973 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:21:28.223949 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b\": container with ID starting with d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b not found: ID does not exist" containerID="d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b" Jul 11 04:21:28.224044 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.223981 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b"} err="failed to get container status \"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b\": rpc error: code = NotFound desc = could not find container \"d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b\": container with ID starting with d6084f2afee54f42228c76198fb8398ed45c40f62bacad09b061df33827c623b not found: ID does not exist" Jul 11 04:21:28.225117 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.225096 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:21:28.230727 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.230704 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e88abb-openshift-default-db5589557-ktmp6"] Jul 11 04:21:28.426572 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:28.426530 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" path="/var/lib/kubelet/pods/3cacb612-c68b-4e20-b0ac-cf94a5435b0a/volumes" Jul 11 04:21:31.302612 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.302567 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.302982 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.302995 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="79544898-0382-41bc-8239-4449f5262a4e" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303010 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303016 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303022 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303028 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303087 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.303115 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.303095 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cacb612-c68b-4e20-b0ac-cf94a5435b0a" containerName="istio-proxy" Jul 11 04:21:31.307648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.307625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.309853 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.309833 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1da175-openshift-default-dockercfg-brt52\"" Jul 11 04:21:31.314545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.314437 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:21:31.377961 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.377926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.377961 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.377963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378227 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378227 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378227 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378227 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksbt\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378383 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378383 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.378466 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.378392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.479817 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.479773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.479817 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.479822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480059 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.479861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480059 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.479876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksbt\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480179 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480179 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480179 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480325 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480793 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480793 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480793 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.480793 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.481094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.480938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.482569 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.482550 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.482725 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.482701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.487318 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.487294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.487626 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.487608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksbt\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt\") pod \"e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.620878 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.620782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:31.767731 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.767702 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:21:31.777184 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:21:31.777140 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1ddb53_8268_48f0_8a1c_198f6c0e8b36.slice/crio-7bf94de9a342a278b09770a7cc164fe57fdc85fe06bc6a30febd84c1f2df9ccb WatchSource:0}: Error finding container 7bf94de9a342a278b09770a7cc164fe57fdc85fe06bc6a30febd84c1f2df9ccb: Status 404 returned error can't find the container with id 7bf94de9a342a278b09770a7cc164fe57fdc85fe06bc6a30febd84c1f2df9ccb Jul 11 04:21:31.781185 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.779894 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:31.781185 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.779993 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:31.781185 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:31.780041 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:32.222404 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:32.222369 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerStarted","Data":"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f"} Jul 11 04:21:32.222587 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:32.222423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerStarted","Data":"7bf94de9a342a278b09770a7cc164fe57fdc85fe06bc6a30febd84c1f2df9ccb"} Jul 11 04:21:32.244311 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:32.244240 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podStartSLOduration=1.2442177939999999 podStartE2EDuration="1.244217794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 04:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:21:32.241069503 +0000 UTC m=+2330.314685785" watchObservedRunningTime="2026-07-11 04:21:32.244217794 +0000 UTC m=+2330.317834057" Jul 11 04:21:32.621842 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:32.621734 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:32.627014 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:32.626990 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:33.227088 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:33.227010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:33.228222 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:33.228202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:39.263092 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.263056 2565 generic.go:356] "Generic (PLEG): container finished" podID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerID="7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458" exitCode=0 Jul 11 04:21:39.263556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.263129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerDied","Data":"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458"} Jul 11 04:21:39.263556 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.263530 2565 scope.go:117] "RemoveContainer" containerID="7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458" Jul 11 04:21:39.264580 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.264553 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:39.264648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.264622 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:39.264704 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:39.264652 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:40.185255 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:40.185221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:40.270545 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:40.270510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerStarted","Data":"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6"} Jul 11 04:21:40.545709 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:40.545587 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:40.546877 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:40.546842 2565 patch_prober.go:28] interesting pod/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 11 04:21:40.547027 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:40.546905 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 11 04:21:41.546277 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:41.545797 2565 patch_prober.go:28] interesting pod/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 11 04:21:41.546277 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:41.545867 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 11 04:21:42.546351 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:42.545863 2565 patch_prober.go:28] interesting pod/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 11 04:21:42.546351 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:42.545931 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 11 04:21:42.675542 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:42.675506 2565 scope.go:117] "RemoveContainer" containerID="c6ed1f33aa53c385540e7a4d57d769b75330d81f65b569a426ec9dea5721fc52" Jul 11 04:21:43.545659 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:43.545616 2565 patch_prober.go:28] interesting pod/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 11 04:21:43.545975 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:43.545945 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 11 04:21:44.559851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:44.559817 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:44.560340 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:44.560300 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:44.561398 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:44.561374 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:21:49.268880 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:49.268835 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 11 04:21:49.269294 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:49.268913 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 11 04:21:50.330796 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.330668 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerID="0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f" exitCode=0 Jul 11 04:21:50.330796 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.330752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerDied","Data":"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f"} Jul 11 04:21:50.331422 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.331285 2565 scope.go:117] "RemoveContainer" containerID="0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f" Jul 11 04:21:50.332369 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.332341 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:50.332463 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.332409 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:50.332463 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:50.332441 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 11 04:21:51.341599 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:51.340651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerStarted","Data":"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c"} Jul 11 04:21:51.620968 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:51.620871 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:51.621261 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:51.621238 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 11 04:21:51.621320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:51.621294 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 11 04:21:52.622131 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:52.622086 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 11 04:21:52.622623 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:52.622154 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 11 04:21:53.621417 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:53.621371 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 11 04:21:53.621694 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:53.621434 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 11 04:21:54.621873 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:54.621833 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 11 04:21:54.622280 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:54.621902 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 11 04:21:55.627445 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:55.625548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:55.627445 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:55.626831 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:21:55.630866 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:21:55.630170 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:22:42.567316 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:22:42.567288 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:22:42.572694 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:22:42.572663 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:27:34.628042 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.627954 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc"] Jul 11 04:27:34.631474 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.631459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:34.633579 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.633564 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hmq47\"" Jul 11 04:27:34.637320 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.637297 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc"] Jul 11 04:27:34.763232 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.763183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc\" (UID: \"6059d2c3-b9f6-468a-a98a-453738a6eda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:34.864822 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.864780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc\" (UID: \"6059d2c3-b9f6-468a-a98a-453738a6eda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:34.865496 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.865475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc\" (UID: \"6059d2c3-b9f6-468a-a98a-453738a6eda6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:34.942718 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:34.942660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:35.081728 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:35.081703 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc"] Jul 11 04:27:35.083959 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:27:35.083930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6059d2c3_b9f6_468a_a98a_453738a6eda6.slice/crio-6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac WatchSource:0}: Error finding container 6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac: Status 404 returned error can't find the container with id 6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac Jul 11 04:27:36.064494 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:36.064404 2565 generic.go:356] "Generic (PLEG): container finished" podID="6059d2c3-b9f6-468a-a98a-453738a6eda6" containerID="a7a1279550c31fd0aa4182e6c41851419c8907efe72e28079f69f5dbd9e120aa" exitCode=0 Jul 11 04:27:36.064903 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:36.064501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" event={"ID":"6059d2c3-b9f6-468a-a98a-453738a6eda6","Type":"ContainerDied","Data":"a7a1279550c31fd0aa4182e6c41851419c8907efe72e28079f69f5dbd9e120aa"} Jul 11 04:27:36.064903 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:36.064562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" event={"ID":"6059d2c3-b9f6-468a-a98a-453738a6eda6","Type":"ContainerStarted","Data":"6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac"} Jul 11 04:27:37.220966 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:37.220939 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:37.287216 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:37.287175 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca\") pod \"6059d2c3-b9f6-468a-a98a-453738a6eda6\" (UID: \"6059d2c3-b9f6-468a-a98a-453738a6eda6\") " Jul 11 04:27:37.287566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:37.287541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6059d2c3-b9f6-468a-a98a-453738a6eda6" (UID: "6059d2c3-b9f6-468a-a98a-453738a6eda6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:27:37.388815 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:37.388778 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059d2c3-b9f6-468a-a98a-453738a6eda6-maas-api-service-ca\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:27:38.081811 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:38.081782 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" Jul 11 04:27:38.081986 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:38.081821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc" event={"ID":"6059d2c3-b9f6-468a-a98a-453738a6eda6","Type":"ContainerDied","Data":"6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac"} Jul 11 04:27:38.081986 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:38.081875 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3bcc004543624b61c61f33463ca4c5995994288ff2f6e1e5933b20b013edac" Jul 11 04:27:42.596466 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:42.596438 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:27:42.602597 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:27:42.602575 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:28:00.401372 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:00.401335 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:28:00.401877 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:00.401556 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" containerID="cri-o://3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6" gracePeriod=30 Jul 11 04:28:05.663287 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.663219 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:28:05.751934 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.751892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.751948 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.751995 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752021 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gc75\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752094 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752138 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752224 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752618 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs\") pod \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\" (UID: \"1cc33284-0ded-4ad4-9e38-8ff45efabf12\") " Jul 11 04:28:05.752618 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752390 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:28:05.752618 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752582 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.752901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752650 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:05.752901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752804 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:05.752901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752812 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:05.752901 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.752845 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:05.754552 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.754512 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75" (OuterVolumeSpecName: "kube-api-access-7gc75") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "kube-api-access-7gc75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:28:05.754552 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.754523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:28:05.754721 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.754576 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:05.754721 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.754711 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cc33284-0ded-4ad4-9e38-8ff45efabf12" (UID: "1cc33284-0ded-4ad4-9e38-8ff45efabf12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:28:05.853352 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853310 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853352 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853345 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gc75\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-kube-api-access-7gc75\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853352 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853354 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853352 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853363 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853621 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853371 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853621 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853378 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853621 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853386 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:05.853621 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:05.853393 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc33284-0ded-4ad4-9e38-8ff45efabf12-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:06.220321 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.220288 2565 generic.go:356] "Generic (PLEG): container finished" podID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerID="3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6" exitCode=0 Jul 11 04:28:06.220495 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.220334 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" Jul 11 04:28:06.220495 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.220358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerDied","Data":"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6"} Jul 11 04:28:06.220495 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.220401 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx" event={"ID":"1cc33284-0ded-4ad4-9e38-8ff45efabf12","Type":"ContainerDied","Data":"dc8df2fec59f933f82e90b587e7bbceb2141ae32fe3e942782bc99e726c57f17"} Jul 11 04:28:06.220495 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.220423 2565 scope.go:117] "RemoveContainer" containerID="3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6" Jul 11 04:28:06.229669 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.229649 2565 scope.go:117] "RemoveContainer" containerID="7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458" Jul 11 04:28:06.240849 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.240822 2565 scope.go:117] "RemoveContainer" containerID="3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6" Jul 11 04:28:06.241176 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:28:06.241148 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6\": container with ID starting with 3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6 not found: ID does not exist" containerID="3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6" Jul 11 04:28:06.241289 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.241186 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6"} err="failed to get container status \"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6\": rpc error: code = NotFound desc = could not find container \"3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6\": container with ID starting with 3dc69c90b7f3de483a0f0f49131c4b3b1ad75ddaec97bc63322992f381d949c6 not found: ID does not exist" Jul 11 04:28:06.241289 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.241210 2565 scope.go:117] "RemoveContainer" containerID="7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458" Jul 11 04:28:06.241531 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:28:06.241511 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458\": container with ID starting with 7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458 not found: ID does not exist" containerID="7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458" Jul 11 04:28:06.241571 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.241538 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458"} err="failed to get container status \"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458\": rpc error: code = NotFound desc = could not find container \"7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458\": container with ID starting with 7055be16bd0ce46dce5f62518d5b941b729132739141057d9bad67a5470be458 not found: ID does not exist" Jul 11 04:28:06.242444 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.242423 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:28:06.245497 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.245474 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6ee78d-openshift-default-6c47c4c9-27ckx"] Jul 11 04:28:06.426578 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:06.426542 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" path="/var/lib/kubelet/pods/1cc33284-0ded-4ad4-9e38-8ff45efabf12/volumes" Jul 11 04:28:26.519414 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:26.519373 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:28:26.519872 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:26.519646 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" containerID="cri-o://aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c" gracePeriod=30 Jul 11 04:28:31.775176 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.775150 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:28:31.903489 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903459 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903527 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903600 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903642 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.903700 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903670 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.904094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903763 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ksbt\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt\") pod \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\" (UID: \"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36\") " Jul 11 04:28:31.904094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:31.904094 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.903955 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:31.904244 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.904136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:31.904244 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.904148 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-credential-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:31.904244 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.904166 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-certs\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:31.904244 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.904179 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 04:28:31.904967 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.904931 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:31.906061 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.906028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 11 04:28:31.906171 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.906070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 04:28:31.906171 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.906084 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt" (OuterVolumeSpecName: "kube-api-access-7ksbt") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "kube-api-access-7ksbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:28:31.906456 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:31.906441 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" (UID: "ee1ddb53-8268-48f0-8a1c-198f6c0e8b36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 04:28:32.004803 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004773 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-envoy\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.004803 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004799 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-token\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.004803 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004810 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istiod-ca-cert\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.005036 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004818 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-podinfo\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.005036 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004825 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-istio-data\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.005036 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004833 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-workload-socket\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.005036 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.004842 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ksbt\" (UniqueName: \"kubernetes.io/projected/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36-kube-api-access-7ksbt\") on node \"ip-10-0-138-240.ec2.internal\" DevicePath \"\"" Jul 11 04:28:32.345070 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.344980 2565 generic.go:356] "Generic (PLEG): container finished" podID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerID="aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c" exitCode=0 Jul 11 04:28:32.345070 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.345033 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" Jul 11 04:28:32.345070 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.345057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerDied","Data":"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c"} Jul 11 04:28:32.345283 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.345117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b" event={"ID":"ee1ddb53-8268-48f0-8a1c-198f6c0e8b36","Type":"ContainerDied","Data":"7bf94de9a342a278b09770a7cc164fe57fdc85fe06bc6a30febd84c1f2df9ccb"} Jul 11 04:28:32.345283 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.345140 2565 scope.go:117] "RemoveContainer" containerID="aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c" Jul 11 04:28:32.354324 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.354305 2565 scope.go:117] "RemoveContainer" containerID="0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f" Jul 11 04:28:32.363022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.363002 2565 scope.go:117] "RemoveContainer" containerID="aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c" Jul 11 04:28:32.363289 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:28:32.363270 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c\": container with ID starting with aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c not found: ID does not exist" containerID="aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c" Jul 11 04:28:32.363332 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.363299 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c"} err="failed to get container status \"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c\": rpc error: code = NotFound desc = could not find container \"aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c\": container with ID starting with aeba3143ab5e24e0f077ec4b883f96d3732437cc98504ce2dceed601a82d195c not found: ID does not exist" Jul 11 04:28:32.363332 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.363318 2565 scope.go:117] "RemoveContainer" containerID="0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f" Jul 11 04:28:32.363529 ip-10-0-138-240 kubenswrapper[2565]: E0711 04:28:32.363513 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f\": container with ID starting with 0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f not found: ID does not exist" containerID="0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f" Jul 11 04:28:32.363579 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.363535 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f"} err="failed to get container status \"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f\": rpc error: code = NotFound desc = could not find container \"0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f\": container with ID starting with 0cbd9f5c756cb0ac2a6a6cfb22b4aeef642ccb16275885fad517b52dbacbcc7f not found: ID does not exist" Jul 11 04:28:32.368167 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.368130 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:28:32.370148 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.370120 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1da175-openshift-default-5fdfc98459-lwd4b"] Jul 11 04:28:32.426699 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.426639 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" path="/var/lib/kubelet/pods/ee1ddb53-8268-48f0-8a1c-198f6c0e8b36/volumes" Jul 11 04:28:32.515040 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:32.515013 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4rqqt_122035e6-6c01-4b22-8d93-68b677a7c69d/server/0.log" Jul 11 04:28:33.002851 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.002824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-scpc2_553f0c20-fdda-4300-a1d1-50869171c0a2/copy-manifests/0.log" Jul 11 04:28:33.009707 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.009666 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-scpc2_553f0c20-fdda-4300-a1d1-50869171c0a2/manager/0.log" Jul 11 04:28:33.611382 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.611355 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29729055-skznz_e90d4763-7d60-43e6-a6d5-1a6b41db89a1/cleanup/0.log" Jul 11 04:28:33.725223 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.725184 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-1c784f07-sd4kr_c77394c3-1ac7-4a3a-9431-6072d918b0ba/revoke-keys/0.log" Jul 11 04:28:33.841477 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.841451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-af4c0929-9sxtx_c0959a69-c706-4f40-9ccc-b1e18302f928/revoke-keys/0.log" Jul 11 04:28:33.951253 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:33.951228 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-20f328-tpp8d_304ae2f0-2622-4ffc-a201-5f008eecfbae/revoke-keys/0.log" Jul 11 04:28:34.168779 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.168751 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-55671289-lxr87_d0f59139-3ea3-48b6-aaeb-9da2ccf2dea1/revoke-keys/0.log" Jul 11 04:28:34.278859 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.278777 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-888089-gw9ck_1f5f9c65-a7c1-4b21-9149-c3d51a722ebf/revoke-keys/0.log" Jul 11 04:28:34.502146 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.502104 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-4b08374c-q2c25_639e63ab-a8d6-4b41-b26c-fb353c99e22d/revoke-keys/0.log" Jul 11 04:28:34.608951 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.608871 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-d7da7c14-l7b6x_068e2f7b-9b93-4b51-988b-04f86cba3eb3/revoke-keys/0.log" Jul 11 04:28:34.715576 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.715551 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-d9f87d6a-htk6l_6cd28145-9761-442d-87ee-01683e92da25/revoke-keys/0.log" Jul 11 04:28:34.827375 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:34.827340 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-6ee78d-pvhrc_6059d2c3-b9f6-468a-a98a-453738a6eda6/revoke-keys/0.log" Jul 11 04:28:35.038703 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:35.038646 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ptw86_dfc9f3a2-b1ea-4cf7-96f5-f911ba8a6309/postgres/0.log" Jul 11 04:28:37.458015 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:37.457988 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cqbnk_0201f665-6c8b-4e08-a54d-83555b2a5f50/istio-proxy/0.log" Jul 11 04:28:37.574654 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:37.574623 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-csxj4_e70c44c2-ff35-4725-9d99-6447d192da92/discovery/0.log" Jul 11 04:28:38.037580 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:38.037551 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-97mdg_4195a265-12ba-472d-baca-fcdd164c577a/payload-pre-processing/0.log" Jul 11 04:28:38.273648 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:38.273620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d95b78bdb-mnxz2_6e4241b4-5633-4ddc-9de4-6873af06759a/router/0.log" Jul 11 04:28:38.737173 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:38.737142 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-sxr7j_b30f0898-2825-47b2-be29-cfbfc0372a82/main/0.log" Jul 11 04:28:45.758816 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:45.758783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bz6lb_2de47295-6cd6-4b77-a057-6d8f5d1b22bd/global-pull-secret-syncer/0.log" Jul 11 04:28:45.905210 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:45.905177 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcvhn_cb06a6fb-10f1-4f74-b5d2-b7c3022068aa/konnectivity-agent/0.log" Jul 11 04:28:45.964053 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:45.964016 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-240.ec2.internal_4f6d6cfb0ec092ad4ea658444bdb52a5/haproxy/0.log" Jul 11 04:28:53.056421 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.056383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-9fhlx_80037351-380c-49a1-b45a-6bd79f190263/cluster-monitoring-operator/0.log" Jul 11 04:28:53.155936 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.155849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57b8fd6656-dfbc2_a45c6c12-eca0-4016-9e1a-be29f330841f/metrics-server/0.log" Jul 11 04:28:53.446813 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.446783 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4bdb_9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4/node-exporter/0.log" Jul 11 04:28:53.480092 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.480054 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4bdb_9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4/kube-rbac-proxy/0.log" Jul 11 04:28:53.502304 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.502277 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4bdb_9bf33ff5-d1b2-4ec5-a3f7-6d2d8ca97df4/init-textfile/0.log" Jul 11 04:28:53.629233 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.629190 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/prometheus/0.log" Jul 11 04:28:53.650727 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.650701 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/config-reloader/0.log" Jul 11 04:28:53.673972 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.673888 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/thanos-sidecar/0.log" Jul 11 04:28:53.693893 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.693858 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/kube-rbac-proxy-web/0.log" Jul 11 04:28:53.716318 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.716282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/kube-rbac-proxy/0.log" Jul 11 04:28:53.737154 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.737119 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/kube-rbac-proxy-thanos/0.log" Jul 11 04:28:53.762107 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:53.762089 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b86b755-2f4b-48f9-98dd-3d24c92c0a56/init-config-reloader/0.log" Jul 11 04:28:56.028071 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.028028 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vsqjk_7a63cb42-0f1c-49bb-926c-79db9f2ab0cd/console-operator/0.log" Jul 11 04:28:56.470178 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470129 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd"] Jul 11 04:28:56.470563 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470545 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" Jul 11 04:28:56.470563 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470563 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470573 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470583 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470593 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470601 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6059d2c3-b9f6-468a-a98a-453738a6eda6" containerName="revoke-keys" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470614 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6059d2c3-b9f6-468a-a98a-453738a6eda6" containerName="revoke-keys" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470700 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470709 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470715 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6059d2c3-b9f6-468a-a98a-453738a6eda6" containerName="revoke-keys" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470720 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc33284-0ded-4ad4-9e38-8ff45efabf12" containerName="istio-proxy" Jul 11 04:28:56.470744 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.470726 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1ddb53-8268-48f0-8a1c-198f6c0e8b36" containerName="istio-proxy" Jul 11 04:28:56.473935 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.473916 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.476072 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.476050 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfvxb\"/\"openshift-service-ca.crt\"" Jul 11 04:28:56.476259 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.476241 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfvxb\"/\"kube-root-ca.crt\"" Jul 11 04:28:56.476993 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.476976 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cfvxb\"/\"default-dockercfg-tsckk\"" Jul 11 04:28:56.480567 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.480545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd"] Jul 11 04:28:56.524642 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.524609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-lib-modules\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.524841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.524710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wjh\" (UniqueName: \"kubernetes.io/projected/be05f156-fcfa-4a54-b925-74378928ca8f-kube-api-access-t4wjh\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.524841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.524738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-podres\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.524841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.524754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-sys\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.524841 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.524777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-proc\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626009 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.625973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wjh\" (UniqueName: \"kubernetes.io/projected/be05f156-fcfa-4a54-b925-74378928ca8f-kube-api-access-t4wjh\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626209 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-podres\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626209 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-sys\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626209 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626118 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-sys\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626209 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-proc\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626209 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-podres\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626454 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-lib-modules\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626454 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626220 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-proc\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.626454 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.626316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05f156-fcfa-4a54-b925-74378928ca8f-lib-modules\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.633861 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.633829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wjh\" (UniqueName: \"kubernetes.io/projected/be05f156-fcfa-4a54-b925-74378928ca8f-kube-api-access-t4wjh\") pod \"perf-node-gather-daemonset-rb4pd\" (UID: \"be05f156-fcfa-4a54-b925-74378928ca8f\") " pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.784897 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.784803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:56.917788 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:56.917743 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd"] Jul 11 04:28:56.920099 ip-10-0-138-240 kubenswrapper[2565]: W0711 04:28:56.920070 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe05f156_fcfa_4a54_b925_74378928ca8f.slice/crio-883c962d12882f9bd6543992a33ff217f1b45874f03a2df6e7c941485131d602 WatchSource:0}: Error finding container 883c962d12882f9bd6543992a33ff217f1b45874f03a2df6e7c941485131d602: Status 404 returned error can't find the container with id 883c962d12882f9bd6543992a33ff217f1b45874f03a2df6e7c941485131d602 Jul 11 04:28:57.461497 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:57.461460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" event={"ID":"be05f156-fcfa-4a54-b925-74378928ca8f","Type":"ContainerStarted","Data":"920ad4e04985c74900d0d6b5349c07da921a70c68ea61313676d72783a347f3b"} Jul 11 04:28:57.461922 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:57.461511 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" event={"ID":"be05f156-fcfa-4a54-b925-74378928ca8f","Type":"ContainerStarted","Data":"883c962d12882f9bd6543992a33ff217f1b45874f03a2df6e7c941485131d602"} Jul 11 04:28:57.461922 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:57.461541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:28:57.475972 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:57.475920 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" podStartSLOduration=1.475905153 podStartE2EDuration="1.475905153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 04:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 04:28:57.474843068 +0000 UTC m=+2775.548459326" watchObservedRunningTime="2026-07-11 04:28:57.475905153 +0000 UTC m=+2775.549521411" Jul 11 04:28:58.199880 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:58.199850 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nn9nn_4a39043c-ddee-4dbc-94b8-d3e387668742/dns/0.log" Jul 11 04:28:58.221639 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:58.221613 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nn9nn_4a39043c-ddee-4dbc-94b8-d3e387668742/kube-rbac-proxy/0.log" Jul 11 04:28:58.331586 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:58.331564 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bn8z_c67a41b4-1865-4ca4-a40c-4bfe16e517b6/dns-node-resolver/0.log" Jul 11 04:28:58.971158 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:58.971128 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6v5k_5eb7690c-2a06-447c-8054-917b81c5ae3d/node-ca/0.log" Jul 11 04:28:59.844351 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:59.844324 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cqbnk_0201f665-6c8b-4e08-a54d-83555b2a5f50/istio-proxy/0.log" Jul 11 04:28:59.968914 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:28:59.968884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-csxj4_e70c44c2-ff35-4725-9d99-6447d192da92/discovery/0.log" Jul 11 04:29:00.104998 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:00.104909 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-97mdg_4195a265-12ba-472d-baca-fcdd164c577a/payload-pre-processing/0.log" Jul 11 04:29:00.157566 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:00.157536 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d95b78bdb-mnxz2_6e4241b4-5633-4ddc-9de4-6873af06759a/router/0.log" Jul 11 04:29:00.861543 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:00.861509 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j5h5d_1d17d352-eb89-4c0e-8b78-71d76d5b567d/serve-healthcheck-canary/0.log" Jul 11 04:29:01.352402 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:01.352371 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-k4bjs_45127447-2a0a-4736-82e3-da2f9aa5b302/insights-operator/0.log" Jul 11 04:29:01.505992 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:01.505966 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrtn_5b491ff8-61f0-426e-adfd-c105a593867b/kube-rbac-proxy/0.log" Jul 11 04:29:01.526355 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:01.526329 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrtn_5b491ff8-61f0-426e-adfd-c105a593867b/exporter/0.log" Jul 11 04:29:01.546790 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:01.546760 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhrtn_5b491ff8-61f0-426e-adfd-c105a593867b/extractor/0.log" Jul 11 04:29:03.476049 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:03.475972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cfvxb/perf-node-gather-daemonset-rb4pd" Jul 11 04:29:03.850514 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:03.850438 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4rqqt_122035e6-6c01-4b22-8d93-68b677a7c69d/server/0.log" Jul 11 04:29:04.106372 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:04.106288 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-scpc2_553f0c20-fdda-4300-a1d1-50869171c0a2/manager/0.log" Jul 11 04:29:04.127099 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:04.127074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766d5cd97-scpc2_553f0c20-fdda-4300-a1d1-50869171c0a2/copy-manifests/0.log" Jul 11 04:29:05.225331 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:05.225303 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d946858c4-mnx4z_6615d324-fc39-4934-98e5-520f16407447/manager/0.log" Jul 11 04:29:06.617867 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:06.617838 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cqbnk_0201f665-6c8b-4e08-a54d-83555b2a5f50/istio-proxy/0.log" Jul 11 04:29:06.727066 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:06.727033 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-csxj4_e70c44c2-ff35-4725-9d99-6447d192da92/discovery/0.log" Jul 11 04:29:06.819729 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:06.819636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-97mdg_4195a265-12ba-472d-baca-fcdd164c577a/payload-pre-processing/0.log" Jul 11 04:29:06.857661 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:06.857607 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d95b78bdb-mnxz2_6e4241b4-5633-4ddc-9de4-6873af06759a/router/0.log" Jul 11 04:29:08.264490 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:08.264462 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cqbnk_0201f665-6c8b-4e08-a54d-83555b2a5f50/istio-proxy/0.log" Jul 11 04:29:08.373336 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:08.373303 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-csxj4_e70c44c2-ff35-4725-9d99-6447d192da92/discovery/0.log" Jul 11 04:29:08.471789 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:08.471760 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-97mdg_4195a265-12ba-472d-baca-fcdd164c577a/payload-pre-processing/0.log" Jul 11 04:29:08.512020 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:08.511982 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d95b78bdb-mnxz2_6e4241b4-5633-4ddc-9de4-6873af06759a/router/0.log" Jul 11 04:29:10.981634 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:10.981605 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-qm9mt_ef37e3bf-38af-4470-87b2-75405cd63f5b/kube-storage-version-migrator-operator/0.log" Jul 11 04:29:12.187152 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.187102 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/kube-multus-additional-cni-plugins/0.log" Jul 11 04:29:12.208854 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.208823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/egress-router-binary-copy/0.log" Jul 11 04:29:12.231535 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.231507 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/cni-plugins/0.log" Jul 11 04:29:12.251043 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.251012 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/bond-cni-plugin/0.log" Jul 11 04:29:12.271647 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.271620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/routeoverride-cni/0.log" Jul 11 04:29:12.295694 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.295660 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/whereabouts-cni-bincopy/0.log" Jul 11 04:29:12.312665 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.312636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lmsx2_d2c499f0-0864-4fea-9232-ad773e09a1d7/whereabouts-cni/0.log" Jul 11 04:29:12.488488 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.488409 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ghkcs_eac0671c-6906-4911-9e68-7067cfba69fe/kube-multus/0.log" Jul 11 04:29:12.568500 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.568474 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2qmkx_e9dec5b8-a00b-4113-9823-1bf997999428/network-metrics-daemon/0.log" Jul 11 04:29:12.586397 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:12.586371 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2qmkx_e9dec5b8-a00b-4113-9823-1bf997999428/kube-rbac-proxy/0.log" Jul 11 04:29:13.665413 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.665383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-controller/0.log" Jul 11 04:29:13.682008 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.681973 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/0.log" Jul 11 04:29:13.682008 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.682010 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovn-acl-logging/1.log" Jul 11 04:29:13.698885 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.698860 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/kube-rbac-proxy-node/0.log" Jul 11 04:29:13.720027 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.720002 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/kube-rbac-proxy-ovn-metrics/0.log" Jul 11 04:29:13.739370 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.739344 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/northd/0.log" Jul 11 04:29:13.763405 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.763379 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/nbdb/0.log" Jul 11 04:29:13.783958 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.783933 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/sbdb/0.log" Jul 11 04:29:13.835022 ip-10-0-138-240 kubenswrapper[2565]: I0711 04:29:13.834993 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz6zf_6177b9e7-f377-426b-9080-5eb8fa3bd5ef/ovnkube-controller/0.log"