Jul 17 12:20:37.995453 ip-10-0-129-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 12:20:37.995611 ip-10-0-129-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 12:20:37.995733 ip-10-0-129-112 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 12:20:37.996059 ip-10-0-129-112 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 12:20:48.086089 ip-10-0-129-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 12:20:48.086109 ip-10-0-129-112 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c976ec2546e44c53889bf608d99cebba -- Jul 17 12:22:29.939030 ip-10-0-129-112 systemd[1]: Starting Kubernetes Kubelet... Jul 17 12:22:30.337919 ip-10-0-129-112 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 12:22:30.337919 ip-10-0-129-112 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 12:22:30.337919 ip-10-0-129-112 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 12:22:30.337919 ip-10-0-129-112 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 12:22:30.337919 ip-10-0-129-112 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 12:22:30.338915 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.338832 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 12:22:30.343222 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343205 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.343222 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343222 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343226 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343229 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343232 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343235 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343238 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343241 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343243 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343246 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343250 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343253 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343261 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343263 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343266 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343269 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343271 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343274 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343276 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343279 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.343289 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343281 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343284 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343286 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343289 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343291 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343294 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343297 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343300 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343303 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343306 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343308 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343310 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343313 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343315 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343318 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343320 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343322 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343325 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343328 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343331 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.343774 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343333 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343335 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343338 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343341 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343346 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343350 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343353 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343355 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343360 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343363 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343365 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343368 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343370 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343373 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343378 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343382 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343384 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343387 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.344292 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343389 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343393 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343396 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343398 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343401 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343403 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343406 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343408 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343411 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343414 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343416 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343418 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343421 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343424 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343426 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343429 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343431 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343434 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343436 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343438 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.344744 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343441 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343443 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343446 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343448 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343450 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343453 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343455 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343457 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343461 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343464 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343467 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343469 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343472 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343474 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343477 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343479 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343482 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343484 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343486 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343489 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.345225 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.343491 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344223 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344230 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344232 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344235 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344238 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344242 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344245 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344247 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344250 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344252 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344257 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344260 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344264 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344267 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344269 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344272 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344274 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344277 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.345734 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344280 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344283 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344286 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344289 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344291 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344294 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344297 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344299 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344302 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344305 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344307 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344310 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344312 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344315 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344317 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344319 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344341 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344344 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344347 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344349 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.346201 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344353 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344356 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344358 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344361 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344363 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344366 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344368 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344371 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344373 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344376 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344378 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344381 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344385 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344388 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344391 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344394 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344396 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344399 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344401 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.346703 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344403 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344407 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344410 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344413 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344416 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344419 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344421 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344424 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344427 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344429 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344432 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344434 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344436 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344439 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344443 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344445 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344447 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344450 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344452 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344455 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.347159 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344457 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344460 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344463 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344467 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344469 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344472 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344475 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344477 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344480 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344482 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344485 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344487 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344489 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344492 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344494 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344496 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344499 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344501 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344504 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344506 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.347643 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344508 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.344510 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345005 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345012 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345015 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345019 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345027 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345030 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345032 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345035 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345038 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345041 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345044 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345046 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345049 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345051 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345056 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345059 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345061 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345064 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.348168 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345066 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345069 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345071 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345075 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345079 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345082 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345085 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345087 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345090 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345093 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345095 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345098 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345101 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345104 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345106 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345109 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345112 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345115 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345118 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.348650 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345127 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345130 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345133 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345136 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345139 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345142 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345145 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345147 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345150 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345155 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345158 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345161 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345163 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345166 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345169 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345172 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345175 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345178 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345180 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345183 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.349164 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345185 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345188 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345190 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345193 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345196 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345200 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345202 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345205 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345207 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345209 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345212 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345215 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345217 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345225 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345228 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345231 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345233 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345237 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345241 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.349645 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345243 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345246 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345250 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345252 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345255 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345257 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345260 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345262 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345264 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345267 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345269 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345272 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345274 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345277 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345279 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345281 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345284 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345286 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345289 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345291 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.350124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345294 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345296 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.345298 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345370 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345377 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345386 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345390 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345409 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345412 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345416 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345421 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345424 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345426 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345429 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345433 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345436 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345439 2568 flags.go:64] FLAG: --cgroup-root="" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345442 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345444 2568 flags.go:64] FLAG: --client-ca-file="" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345447 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345450 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345457 2568 flags.go:64] FLAG: --cluster-domain="" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345460 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345463 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 12:22:30.350628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345466 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345469 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345473 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345475 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345479 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345482 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345485 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345488 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345491 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345493 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345496 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345500 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345503 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345505 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345508 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345511 2568 flags.go:64] FLAG: --enable-server="true" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345522 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345529 2568 flags.go:64] FLAG: --event-burst="100" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345532 2568 flags.go:64] FLAG: --event-qps="50" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345534 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345537 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345541 2568 flags.go:64] FLAG: --eviction-hard="" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345544 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345547 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345550 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 12:22:30.351229 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345553 2568 flags.go:64] FLAG: --eviction-soft="" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345555 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345566 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345569 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345572 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345575 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345578 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345580 2568 flags.go:64] FLAG: --feature-gates="" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345584 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345587 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345589 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345592 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345596 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345599 2568 flags.go:64] FLAG: --help="false" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345602 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345605 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345608 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345611 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345614 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345617 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345620 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345623 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345625 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 12:22:30.351844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345628 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345636 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345640 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345642 2568 flags.go:64] FLAG: --kube-reserved="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345645 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345647 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345650 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345653 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345655 2568 flags.go:64] FLAG: --lock-file="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345658 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345661 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345663 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345668 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345671 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345674 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345676 2568 flags.go:64] FLAG: --logging-format="text" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345694 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345698 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345701 2568 flags.go:64] FLAG: --manifest-url="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345703 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345708 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345711 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345715 2568 flags.go:64] FLAG: --max-pods="110" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345717 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345720 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 12:22:30.352466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345723 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345726 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345728 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345731 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345734 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345741 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345744 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345747 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345750 2568 flags.go:64] FLAG: --pod-cidr="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345758 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345764 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345767 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345770 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345773 2568 flags.go:64] FLAG: --port="10250" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345777 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345779 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051b20645958423f1" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345782 2568 flags.go:64] FLAG: --qos-reserved="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345785 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345788 2568 flags.go:64] FLAG: --register-node="true" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345791 2568 flags.go:64] FLAG: --register-with-taints="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345794 2568 flags.go:64] FLAG: --registry-burst="10" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345797 2568 flags.go:64] FLAG: --registry-qps="5" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345800 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345802 2568 flags.go:64] FLAG: --reserved-memory="" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345806 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 12:22:30.353090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345808 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345811 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345814 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345816 2568 flags.go:64] FLAG: --runonce="false" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345821 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345823 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345826 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345829 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345832 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345834 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345837 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345840 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345843 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345846 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345848 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345851 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345854 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345862 2568 flags.go:64] FLAG: --system-cgroups="" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345865 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345870 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345873 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345877 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345884 2568 flags.go:64] FLAG: --tls-min-version="" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345887 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345890 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 12:22:30.353822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345893 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345895 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345898 2568 flags.go:64] FLAG: --v="2" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345902 2568 flags.go:64] FLAG: --version="false" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345906 2568 flags.go:64] FLAG: --vmodule="" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345909 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.345912 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346076 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346080 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346083 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346086 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346090 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346093 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346096 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346099 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346101 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346104 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346107 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346109 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346112 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346114 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.354468 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346117 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346120 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346122 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346124 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346133 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346135 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346138 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346142 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346144 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346147 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346149 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346152 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346154 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346157 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346159 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346162 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346164 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346166 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346169 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.355042 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346172 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346176 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346179 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346183 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346185 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346188 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346191 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346193 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346196 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346198 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346202 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346205 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346208 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346211 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346214 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346217 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346219 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346222 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346230 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.355502 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346233 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346237 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346240 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346242 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346245 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346248 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346252 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346254 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346257 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346259 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346262 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346264 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346267 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346269 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346271 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346274 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346277 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346280 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346282 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346285 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.355992 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346287 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346290 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346292 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346294 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346297 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346299 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346301 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346304 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346306 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346309 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346311 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346314 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346322 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346325 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346328 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346330 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346333 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346335 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346338 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346340 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.356484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346343 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346345 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346347 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346350 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346352 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346355 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.346357 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.346364 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.353526 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.353540 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.354350 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.354359 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354421 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354426 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.356984 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354429 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354433 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354436 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354439 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354442 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354445 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354449 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354453 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354457 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354460 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354464 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354467 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354471 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354474 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354477 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354479 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354482 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354485 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354488 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.357330 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354490 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354493 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354495 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354498 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354500 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354503 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354506 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354508 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354512 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354515 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354517 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354520 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354522 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354525 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354527 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354530 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354532 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354535 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354537 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354541 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.357847 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354544 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354547 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354550 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354552 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354555 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354558 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354561 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354563 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354566 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354568 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354571 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354573 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354576 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354578 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354581 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354583 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354585 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354588 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354590 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354593 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.358338 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354596 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354599 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354602 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354604 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354607 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354609 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354612 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354614 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354616 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354619 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354621 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354623 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354626 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354628 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354630 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354633 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354635 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354638 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354641 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354643 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.358914 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354645 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354648 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354650 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354653 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354655 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354657 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354660 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354662 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354665 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354667 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354669 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354672 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354675 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354678 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354680 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354695 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354697 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.359458 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354700 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.354705 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354819 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354824 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354827 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354830 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354832 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354835 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354838 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354841 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354843 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354845 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354848 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354851 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354853 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 12:22:30.359994 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354856 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354858 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354861 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354863 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354866 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354869 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354871 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354874 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354876 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354878 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354881 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354883 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354886 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354888 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354891 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354893 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354896 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354899 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354901 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354903 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 12:22:30.360365 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354906 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354908 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354911 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354913 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354916 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354919 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354923 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354925 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354929 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354932 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354936 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354939 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354942 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354944 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354947 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354949 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354952 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354954 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354957 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 12:22:30.360894 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354959 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354962 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354964 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354967 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354969 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354972 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354975 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354977 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354979 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354982 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354984 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354987 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354989 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354992 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354994 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.354997 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355000 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355002 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355005 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 12:22:30.361354 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355007 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355010 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355012 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355015 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355017 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355019 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355025 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355028 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355030 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355033 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355035 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355038 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355040 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355043 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355045 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355048 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355051 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355054 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355057 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355059 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 12:22:30.361921 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355062 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355065 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355067 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355070 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355072 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355074 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355077 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:30.355079 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.355084 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.355196 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.357557 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.358997 2568 server.go:1019] "Starting client certificate rotation" Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.359120 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 12:22:30.362407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.359808 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 12:22:30.362775 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.361563 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 12:22:30.364887 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.364854 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 12:22:30.377480 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.377453 2568 log.go:25] "Validated CRI v1 runtime API" Jul 17 12:22:30.381190 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.381173 2568 log.go:25] "Validated CRI v1 image API" Jul 17 12:22:30.382061 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.382038 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 12:22:30.385271 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.385243 2568 fs.go:135] Filesystem UUIDs: map[0d4e0d6b-71dd-4597-bb7e-468c3973d393:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d6363fc9-51af-4614-ab84-f21428ef4cc1:/dev/nvme0n1p3] Jul 17 12:22:30.385353 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.385271 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 12:22:30.390101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.389988 2568 manager.go:216] Machine: {Timestamp:2026-07-17 12:22:30.388840785 +0000 UTC m=+0.333700280 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100074 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aec66ed3e0c3646720274fa2362b6 SystemUUID:ec2aec66-ed3e-0c36-4672-0274fa2362b6 BootID:c976ec25-46e4-4c53-889b-f608d99cebba Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:4e:a6:32:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:4e:a6:32:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:3a:ec:a7:ea:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 12:22:30.390101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.390093 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 12:22:30.390220 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.390205 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 12:22:30.390901 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.390886 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 12:22:30.393160 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.393137 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 12:22:30.393296 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.393165 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 12:22:30.393885 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.393875 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 12:22:30.393917 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.393887 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 12:22:30.393917 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.393895 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 12:22:30.394693 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.394671 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 12:22:30.396571 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.396561 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 12:22:30.400167 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400154 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 17 12:22:30.400289 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400279 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 12:22:30.400325 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400311 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 17 12:22:30.400325 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400321 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 12:22:30.400397 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400334 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 12:22:30.400397 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400344 2568 kubelet.go:398] "Adding apiserver pod source" Jul 17 12:22:30.400397 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400354 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 12:22:30.400729 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400716 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 12:22:30.400795 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.400734 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 12:22:30.405903 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.405874 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 12:22:30.408157 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.408144 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 12:22:30.408211 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.408175 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 12:22:30.410497 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410485 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 12:22:30.410534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410503 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 12:22:30.410534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410510 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 12:22:30.410534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410515 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 12:22:30.410534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410524 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 12:22:30.410534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410529 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410537 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410543 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410550 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410555 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410563 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 12:22:30.410657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.410568 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 12:22:30.411207 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.411198 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 12:22:30.411207 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.411208 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 12:22:30.415547 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.415534 2568 server.go:1268] "Started kubelet" Jul 17 12:22:30.415798 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.415651 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 12:22:30.419265 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.415628 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 12:22:30.419265 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.419180 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 12:22:30.419376 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.419319 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xlf2" Jul 17 12:22:30.419887 ip-10-0-129-112 systemd[1]: Started Kubernetes Kubelet. Jul 17 12:22:30.421349 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.421335 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 12:22:30.422198 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.422183 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 17 12:22:30.422716 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.422677 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xlf2" Jul 17 12:22:30.424263 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.424115 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 12:22:30.424786 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.424754 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 12:22:30.424860 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.424775 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 12:22:30.426510 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.426491 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 12:22:30.427046 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.425903 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-112.ec2.internal.18c3132a7375d371 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-112.ec2.internal,UID:ip-10-0-129-112.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-112.ec2.internal,},FirstTimestamp:2026-07-17 12:22:30.415504241 +0000 UTC m=+0.360363736,LastTimestamp:2026-07-17 12:22:30.415504241 +0000 UTC m=+0.360363736,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-112.ec2.internal,}" Jul 17 12:22:30.432625 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.432608 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 12:22:30.432719 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.432623 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 12:22:30.432719 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.432712 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.433277 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433262 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 12:22:30.433277 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433264 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 12:22:30.433418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433287 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 12:22:30.433418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433356 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 17 12:22:30.433418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433362 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 17 12:22:30.433923 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433902 2568 factory.go:55] Registering systemd factory Jul 17 12:22:30.434007 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.433926 2568 factory.go:223] Registration of the systemd container factory successfully Jul 17 12:22:30.434159 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434145 2568 factory.go:153] Registering CRI-O factory Jul 17 12:22:30.434233 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434162 2568 factory.go:223] Registration of the crio container factory successfully Jul 17 12:22:30.434233 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434218 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 12:22:30.434344 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434243 2568 factory.go:103] Registering Raw factory Jul 17 12:22:30.434344 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434259 2568 manager.go:1196] Started watching for new ooms in manager Jul 17 12:22:30.434607 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434590 2568 manager.go:349] Starting recovery of all containers Jul 17 12:22:30.434938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.434798 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:30.437444 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.437421 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-112.ec2.internal\" not found" node="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.443580 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.443563 2568 manager.go:354] Recovery completed Jul 17 12:22:30.447561 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.447548 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.449462 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.449447 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.449528 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.449479 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.449528 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.449489 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.450035 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450023 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 12:22:30.450035 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450034 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 12:22:30.450140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450043 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.450140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450050 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 17 12:22:30.450783 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450764 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.450875 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450796 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.450875 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.450807 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.452192 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.452181 2568 policy_none.go:49] "None policy: Start" Jul 17 12:22:30.452227 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.452195 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 12:22:30.452227 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.452205 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 12:22:30.453853 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.453843 2568 policy_none.go:47] "Start" Jul 17 12:22:30.496476 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496328 2568 manager.go:337] "Starting Device Plugin manager" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.496517 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496532 2568 server.go:87] "Starting device plugin registration server" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496800 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496810 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496898 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496988 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.496999 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.497634 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 12:22:30.509201 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.497676 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.578934 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.578899 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 12:22:30.580288 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.580266 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 12:22:30.580410 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.580302 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 12:22:30.580410 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.580325 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 12:22:30.580410 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.580356 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 12:22:30.582666 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.582649 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:30.597802 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.597721 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.598712 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.598678 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.598785 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.598730 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.598785 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.598742 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.598785 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.598773 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.606651 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.606636 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.606718 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.606660 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-112.ec2.internal\": node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.615123 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.615104 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.681202 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.681170 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal"] Jul 17 12:22:30.681279 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.681262 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.682964 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.682947 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.683043 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.682987 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.683043 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.683003 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.684327 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.684316 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.684449 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.684427 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.684484 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.684466 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.685088 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685070 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.685171 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685098 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.685171 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685108 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.685171 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685144 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.685171 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685161 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.685171 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.685172 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.686456 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.686443 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.686543 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.686468 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 12:22:30.688123 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.688104 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientMemory" Jul 17 12:22:30.688201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.688141 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 12:22:30.688201 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.688161 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeHasSufficientPID" Jul 17 12:22:30.704411 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.704383 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-112.ec2.internal\" not found" node="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.708605 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.708587 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-112.ec2.internal\" not found" node="ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.715752 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.715728 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.816297 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.816258 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.834611 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.834584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.834718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.834619 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.834718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.834638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e412b5125944006d4709c364d63da6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-112.ec2.internal\" (UID: \"b9e412b5125944006d4709c364d63da6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.917039 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:30.916957 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:30.935345 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.935459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.935459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e412b5125944006d4709c364d63da6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-112.ec2.internal\" (UID: \"b9e412b5125944006d4709c364d63da6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.935459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.935459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935439 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f370020aead5ae2f513b709de0edb00e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal\" (UID: \"f370020aead5ae2f513b709de0edb00e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:30.935459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:30.935446 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e412b5125944006d4709c364d63da6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-112.ec2.internal\" (UID: \"b9e412b5125944006d4709c364d63da6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:31.008454 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.008411 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:31.011581 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.011559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:31.018028 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:31.018010 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:31.118720 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:31.118654 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:31.219302 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:31.219204 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:31.319773 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:31.319731 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-112.ec2.internal\" not found" Jul 17 12:22:31.358258 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.358228 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 12:22:31.358771 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.358384 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 12:22:31.358771 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.358432 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 12:22:31.416373 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.416344 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:31.429048 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.429008 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 12:17:30 +0000 UTC" deadline="2027-12-12 02:22:14.789247977 +0000 UTC" Jul 17 12:22:31.429048 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.429040 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12301h59m43.360212586s" Jul 17 12:22:31.433761 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.433729 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" Jul 17 12:22:31.433761 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.433754 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 12:22:31.443501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.443466 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 12:22:31.445600 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.445583 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 12:22:31.447645 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.447624 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" Jul 17 12:22:31.455972 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.455952 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 12:22:31.469071 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.469034 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjd58" Jul 17 12:22:31.474962 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.474906 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jjd58" Jul 17 12:22:31.594726 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:31.594695 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e412b5125944006d4709c364d63da6.slice/crio-75d8b450b3400322fd4efee4da974e3339c55953d9e6ba2d3a484b1865558fd0 WatchSource:0}: Error finding container 75d8b450b3400322fd4efee4da974e3339c55953d9e6ba2d3a484b1865558fd0: Status 404 returned error can't find the container with id 75d8b450b3400322fd4efee4da974e3339c55953d9e6ba2d3a484b1865558fd0 Jul 17 12:22:31.595296 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:31.595278 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf370020aead5ae2f513b709de0edb00e.slice/crio-2aa43fe1ed49a212c4e93570ef35907de0287eae5a933115daaf03734bfd6da4 WatchSource:0}: Error finding container 2aa43fe1ed49a212c4e93570ef35907de0287eae5a933115daaf03734bfd6da4: Status 404 returned error can't find the container with id 2aa43fe1ed49a212c4e93570ef35907de0287eae5a933115daaf03734bfd6da4 Jul 17 12:22:31.601336 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.601301 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 12:22:31.775910 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:31.774463 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:32.400711 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.400591 2568 apiserver.go:52] "Watching apiserver" Jul 17 12:22:32.407092 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.407062 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 12:22:32.407528 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.407496 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2tqwc","openshift-multus/multus-sz2nb","openshift-network-diagnostics/network-check-target-sp68r","openshift-network-operator/iptables-alerter-487k8","kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal","openshift-dns/node-resolver-2nfht","openshift-image-registry/node-ca-mvl9t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal","openshift-multus/multus-additional-cni-plugins-6v7ml","openshift-multus/network-metrics-daemon-l6t2j","openshift-ovn-kubernetes/ovnkube-node-png4r","kube-system/global-pull-secret-syncer-2n479","kube-system/konnectivity-agent-cp7zn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb"] Jul 17 12:22:32.409514 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.409492 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:32.409637 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.409572 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:32.410923 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.410894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.412381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.412365 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.413398 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.413376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 12:22:32.413603 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.413588 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cpk4q\"" Jul 17 12:22:32.413914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.413893 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 12:22:32.414094 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414077 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 12:22:32.414166 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414096 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414327 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414340 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414526 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414880 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.414919 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.415109 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w54p6\"" Jul 17 12:22:32.415657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.415415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.416341 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.416317 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.416648 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.416627 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxvln\"" Jul 17 12:22:32.416819 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.416763 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.416897 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.416865 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 12:22:32.417288 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.417261 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.417533 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.417511 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rfzz6\"" Jul 17 12:22:32.417607 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.417548 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.417664 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.417512 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.418459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.418442 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.419159 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.418987 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.419159 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.419126 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.419330 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.419240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qfspz\"" Jul 17 12:22:32.420352 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.420333 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.420448 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.420410 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:32.420860 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.420637 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 12:22:32.420860 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.420659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7xkpc\"" Jul 17 12:22:32.422144 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.422125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.424369 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.424106 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 12:22:32.424369 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.424136 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 12:22:32.424575 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.424559 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bpjc7\"" Jul 17 12:22:32.424662 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.424612 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 12:22:32.424826 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.424807 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.424912 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.424883 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:32.425095 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.425073 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 12:22:32.425190 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.425102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.429588 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.426863 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.429588 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.428058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.429588 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.428785 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.430628 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.430586 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 12:22:32.430817 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.430796 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cf6hz\"" Jul 17 12:22:32.431322 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.431299 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2zvf4\"" Jul 17 12:22:32.431412 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.431346 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 12:22:32.431551 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.431527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 12:22:32.432177 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.432156 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 12:22:32.432384 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.432365 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 12:22:32.434992 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.434971 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 12:22:32.443916 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.443894 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.444044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.443925 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.444044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.443952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.444044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.443977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.444044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444091 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovn-node-metrics-cert\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4jt\" (UniqueName: \"kubernetes.io/projected/61b4f3f4-920d-417f-9cf0-d3d60b64a434-kube-api-access-nd4jt\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-env-overrides\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:32.444275 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtkg\" (UniqueName: \"kubernetes.io/projected/5290d129-15a0-4b75-b59a-3878bf3e9e7b-kube-api-access-fjtkg\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-netns\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-bin\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-netd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85r7\" (UniqueName: \"kubernetes.io/projected/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kube-api-access-s85r7\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrkk\" (UniqueName: \"kubernetes.io/projected/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-kube-api-access-hrrkk\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-system-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cnibin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-run\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.444574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444537 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-var-lib-kubelet\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-konnectivity-ca\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-slash\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-systemd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-sys-fs\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-modprobe-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-conf\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444830 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpd4\" (UniqueName: \"kubernetes.io/projected/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-kube-api-access-fjpd4\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-log-socket\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-socket-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.444983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-dbus\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.445075 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-host-slash\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-bin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-hostroot\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-sys\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-lib-modules\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-host\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b4f3f4-920d-417f-9cf0-d3d60b64a434-host\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-multus\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59fj\" (UniqueName: \"kubernetes.io/projected/f3f71dfb-6002-40d1-8004-9e57f9bcf384-kube-api-access-f59fj\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-daemon-config\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gx9\" (UniqueName: \"kubernetes.io/projected/49468ee0-4570-43d8-b56b-ca2fb6ef2379-kube-api-access-m4gx9\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-netns\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-var-lib-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cnibin\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cni-binary-copy\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445512 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-socket-dir-parent\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.445718 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-multus-certs\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-ovn\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-config\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-script-lib\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445648 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-kubernetes\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-systemd\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445735 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-tmp\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-iptables-alerter-script\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445853 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5290d129-15a0-4b75-b59a-3878bf3e9e7b-hosts-file\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-node-log\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-registration-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.445976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-system-cni-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-os-release\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-conf-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.446613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-os-release\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysconfig\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5290d129-15a0-4b75-b59a-3878bf3e9e7b-tmp-dir\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-k8s-cni-cncf-io\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-etc-kubernetes\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-kubelet\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-systemd-units\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446339 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-etc-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-agent-certs\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446397 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-device-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-kubelet-config\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrr5\" (UniqueName: \"kubernetes.io/projected/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-kube-api-access-ctrr5\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446502 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-tuned\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pk7\" (UniqueName: \"kubernetes.io/projected/0281067f-af52-408c-8faa-6c4d2afe3b74-kube-api-access-l5pk7\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b4f3f4-920d-417f-9cf0-d3d60b64a434-serviceca\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.447381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.446596 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-kubelet\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.478685 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.478006 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:31 +0000 UTC" deadline="2028-04-21 04:45:47.74032497 +0000 UTC" Jul 17 12:22:32.478685 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.478051 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15448h23m15.262277876s" Jul 17 12:22:32.547339 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-env-overrides\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtkg\" (UniqueName: \"kubernetes.io/projected/5290d129-15a0-4b75-b59a-3878bf3e9e7b-kube-api-access-fjtkg\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-netns\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-bin\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-netd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547534 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85r7\" (UniqueName: \"kubernetes.io/projected/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kube-api-access-s85r7\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrkk\" (UniqueName: \"kubernetes.io/projected/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-kube-api-access-hrrkk\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-system-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cnibin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-run\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547686 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-var-lib-kubelet\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-konnectivity-ca\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-slash\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-systemd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-sys-fs\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-modprobe-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-conf\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.547913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpd4\" (UniqueName: \"kubernetes.io/projected/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-kube-api-access-fjpd4\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-log-socket\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.547994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-socket-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-dbus\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-host-slash\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-bin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-hostroot\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-sys\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-lib-modules\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-host\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b4f3f4-920d-417f-9cf0-d3d60b64a434-host\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-multus\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548298 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59fj\" (UniqueName: \"kubernetes.io/projected/f3f71dfb-6002-40d1-8004-9e57f9bcf384-kube-api-access-f59fj\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-daemon-config\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gx9\" (UniqueName: \"kubernetes.io/projected/49468ee0-4570-43d8-b56b-ca2fb6ef2379-kube-api-access-m4gx9\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.548509 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-netns\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-var-lib-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cnibin\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cni-binary-copy\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-socket-dir-parent\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548570 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-multus-certs\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-ovn\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-config\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548652 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-script-lib\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-kubernetes\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-systemd\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548753 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-tmp\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-iptables-alerter-script\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5290d129-15a0-4b75-b59a-3878bf3e9e7b-hosts-file\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-node-log\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-registration-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-system-cni-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-os-release\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-conf-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.548980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-os-release\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysconfig\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5290d129-15a0-4b75-b59a-3878bf3e9e7b-tmp-dir\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-k8s-cni-cncf-io\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-etc-kubernetes\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-kubelet\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-systemd-units\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549166 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-etc-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-agent-certs\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-device-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-kubelet-config\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrr5\" (UniqueName: \"kubernetes.io/projected/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-kube-api-access-ctrr5\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-tuned\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.549864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pk7\" (UniqueName: \"kubernetes.io/projected/0281067f-af52-408c-8faa-6c4d2afe3b74-kube-api-access-l5pk7\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b4f3f4-920d-417f-9cf0-d3d60b64a434-serviceca\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-kubelet\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovn-node-metrics-cert\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.550477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.549630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4jt\" (UniqueName: \"kubernetes.io/projected/61b4f3f4-920d-417f-9cf0-d3d60b64a434-kube-api-access-nd4jt\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.550877 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.550529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-env-overrides\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.550914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.550887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-run\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.550914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.550900 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-socket-dir-parent\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.550993 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.550885 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-socket-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.550993 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.550925 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-log-socket\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551179 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-systemd-units\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551278 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-multus-certs\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.551278 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551220 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-ovn\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551391 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551350 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-kubelet\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551479 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551436 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 12:22:32.551538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.551538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-kubelet\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.551640 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551581 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-kubelet-config\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.551640 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551626 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-device-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.551640 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.551829 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-etc-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551829 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-config\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.551829 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551736 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.551829 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.551797 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:32.552050 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.551898 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:33.051856748 +0000 UTC m=+2.996716232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:32.552148 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.552204 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.551799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.552257 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovnkube-script-lib\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.552660 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5290d129-15a0-4b75-b59a-3878bf3e9e7b-hosts-file\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-var-lib-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552774 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-run-netns\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-node-log\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552825 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cnibin\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-registration-dir\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.553037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.552912 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-daemon-config\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553044 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-system-cni-dir\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-dbus\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553087 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-host-slash\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-netns\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-bin\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-kubernetes\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-os-release\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553181 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-hostroot\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553219 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-bin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-systemd\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553331 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-conf-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553354 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-lib-modules\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553360 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-openvswitch\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-var-lib-kubelet\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553408 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-host\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553426 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-sys\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553463 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-os-release\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.553591 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553498 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-cni-netd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553540 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-system-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysconfig\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553622 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cnibin\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-konnectivity-ca\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.553995 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5290d129-15a0-4b75-b59a-3878bf3e9e7b-tmp-dir\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554030 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-host-slash\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554057 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-run-k8s-cni-cncf-io\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554070 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61b4f3f4-920d-417f-9cf0-d3d60b64a434-host\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-host-var-lib-cni-multus\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/49468ee0-4570-43d8-b56b-ca2fb6ef2379-run-systemd\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554216 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554271 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-sys-fs\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-modprobe-d\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.554598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.554600 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.555451 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555429 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/49468ee0-4570-43d8-b56b-ca2fb6ef2379-ovn-node-metrics-cert\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.555518 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a-agent-certs\") pod \"konnectivity-agent-cp7zn\" (UID: \"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a\") " pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.555570 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555512 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-etc-kubernetes\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.555651 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555628 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61b4f3f4-920d-417f-9cf0-d3d60b64a434-serviceca\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.555735 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555712 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3f71dfb-6002-40d1-8004-9e57f9bcf384-multus-cni-dir\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.555826 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.555811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-sysctl-conf\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.555901 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.555885 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:32.555954 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.555949 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:22:33.055934589 +0000 UTC m=+3.000794073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:32.556165 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.556055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3f71dfb-6002-40d1-8004-9e57f9bcf384-cni-binary-copy\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.558172 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.558149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-etc-tuned\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.558172 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.558161 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-iptables-alerter-script\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.558491 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.558470 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0281067f-af52-408c-8faa-6c4d2afe3b74-tmp\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.564287 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.564260 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4jt\" (UniqueName: \"kubernetes.io/projected/61b4f3f4-920d-417f-9cf0-d3d60b64a434-kube-api-access-nd4jt\") pod \"node-ca-mvl9t\" (UID: \"61b4f3f4-920d-417f-9cf0-d3d60b64a434\") " pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.566373 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.566303 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:32.566373 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.566332 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:32.566373 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.566348 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:32.566702 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:32.566414 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:33.06639249 +0000 UTC m=+3.011251975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:32.567712 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.567685 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pk7\" (UniqueName: \"kubernetes.io/projected/0281067f-af52-408c-8faa-6c4d2afe3b74-kube-api-access-l5pk7\") pod \"tuned-2tqwc\" (UID: \"0281067f-af52-408c-8faa-6c4d2afe3b74\") " pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.571204 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.571174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85r7\" (UniqueName: \"kubernetes.io/projected/82c3e75d-af2b-4e7d-bac9-c4a3b89566cf-kube-api-access-s85r7\") pod \"aws-ebs-csi-driver-node-8r8lb\" (UID: \"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.571492 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.571469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtkg\" (UniqueName: \"kubernetes.io/projected/5290d129-15a0-4b75-b59a-3878bf3e9e7b-kube-api-access-fjtkg\") pod \"node-resolver-2nfht\" (UID: \"5290d129-15a0-4b75-b59a-3878bf3e9e7b\") " pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.571930 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.571910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrkk\" (UniqueName: \"kubernetes.io/projected/b579fbbe-8aa3-476b-8895-6ad0daddd5a6-kube-api-access-hrrkk\") pod \"iptables-alerter-487k8\" (UID: \"b579fbbe-8aa3-476b-8895-6ad0daddd5a6\") " pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.572179 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.572155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpd4\" (UniqueName: \"kubernetes.io/projected/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-kube-api-access-fjpd4\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:32.572273 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.572156 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gx9\" (UniqueName: \"kubernetes.io/projected/49468ee0-4570-43d8-b56b-ca2fb6ef2379-kube-api-access-m4gx9\") pod \"ovnkube-node-png4r\" (UID: \"49468ee0-4570-43d8-b56b-ca2fb6ef2379\") " pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.572328 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.572282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59fj\" (UniqueName: \"kubernetes.io/projected/f3f71dfb-6002-40d1-8004-9e57f9bcf384-kube-api-access-f59fj\") pod \"multus-sz2nb\" (UID: \"f3f71dfb-6002-40d1-8004-9e57f9bcf384\") " pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.577194 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.577167 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrr5\" (UniqueName: \"kubernetes.io/projected/ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0-kube-api-access-ctrr5\") pod \"multus-additional-cni-plugins-6v7ml\" (UID: \"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0\") " pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.586047 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.585980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" event={"ID":"b9e412b5125944006d4709c364d63da6","Type":"ContainerStarted","Data":"75d8b450b3400322fd4efee4da974e3339c55953d9e6ba2d3a484b1865558fd0"} Jul 17 12:22:32.587370 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.587337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" event={"ID":"f370020aead5ae2f513b709de0edb00e","Type":"ContainerStarted","Data":"2aa43fe1ed49a212c4e93570ef35907de0287eae5a933115daaf03734bfd6da4"} Jul 17 12:22:32.727861 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.727773 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" Jul 17 12:22:32.734986 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.734959 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvl9t" Jul 17 12:22:32.738184 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.738165 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:32.747293 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.747270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-487k8" Jul 17 12:22:32.754835 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.754813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2nfht" Jul 17 12:22:32.762245 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.762225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" Jul 17 12:22:32.771086 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.771067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz2nb" Jul 17 12:22:32.786311 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.786278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:32.793756 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.793737 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:32.800903 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.800871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" Jul 17 12:22:32.854578 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:32.854545 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 12:22:33.054609 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.054517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:33.054827 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.054726 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:33.054827 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.054809 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:34.05478552 +0000 UTC m=+3.999645010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:33.155593 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.155537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:33.155593 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.155601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155748 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155748 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155778 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155791 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155804 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:22:34.155789805 +0000 UTC m=+4.100649303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:33.155859 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:33.155835 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:34.155820884 +0000 UTC m=+4.100680369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:33.478703 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.478556 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:31 +0000 UTC" deadline="2027-12-19 10:49:38.527392361 +0000 UTC" Jul 17 12:22:33.478703 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.478627 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12478h27m5.048770163s" Jul 17 12:22:33.581020 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.580901 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb579fbbe_8aa3_476b_8895_6ad0daddd5a6.slice/crio-f6e0af85aa7ba0721da9b5e287aa633c2cbc89cc12d401fccb055db5b10b0009 WatchSource:0}: Error finding container f6e0af85aa7ba0721da9b5e287aa633c2cbc89cc12d401fccb055db5b10b0009: Status 404 returned error can't find the container with id f6e0af85aa7ba0721da9b5e287aa633c2cbc89cc12d401fccb055db5b10b0009 Jul 17 12:22:33.581771 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.581729 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded35a9c8_6dc7_4572_b4ed_e5bb60a76ac0.slice/crio-91c20deddf53b4a5141320fb1edc382742d6d1cf49a1d6d6cd7f88245548ec2b WatchSource:0}: Error finding container 91c20deddf53b4a5141320fb1edc382742d6d1cf49a1d6d6cd7f88245548ec2b: Status 404 returned error can't find the container with id 91c20deddf53b4a5141320fb1edc382742d6d1cf49a1d6d6cd7f88245548ec2b Jul 17 12:22:33.583857 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.583832 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4daf38fd_9b5a_49b7_b1a7_87f8c365bb4a.slice/crio-684a8e32f7ad166ba80d5f49d8af216fd1363e93bdb73a04b3eb9d93499808cb WatchSource:0}: Error finding container 684a8e32f7ad166ba80d5f49d8af216fd1363e93bdb73a04b3eb9d93499808cb: Status 404 returned error can't find the container with id 684a8e32f7ad166ba80d5f49d8af216fd1363e93bdb73a04b3eb9d93499808cb Jul 17 12:22:33.586988 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.586959 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b4f3f4_920d_417f_9cf0_d3d60b64a434.slice/crio-fe38628a7b7f34c5bfd811f4546a4e32b4712fe235c904e845913269ed4c90bb WatchSource:0}: Error finding container fe38628a7b7f34c5bfd811f4546a4e32b4712fe235c904e845913269ed4c90bb: Status 404 returned error can't find the container with id fe38628a7b7f34c5bfd811f4546a4e32b4712fe235c904e845913269ed4c90bb Jul 17 12:22:33.588138 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.587843 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f71dfb_6002_40d1_8004_9e57f9bcf384.slice/crio-4a423767b477512e2643bcce4dd001e8eeca05e7480ce46cebeb758346d8f989 WatchSource:0}: Error finding container 4a423767b477512e2643bcce4dd001e8eeca05e7480ce46cebeb758346d8f989: Status 404 returned error can't find the container with id 4a423767b477512e2643bcce4dd001e8eeca05e7480ce46cebeb758346d8f989 Jul 17 12:22:33.588723 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.588695 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0281067f_af52_408c_8faa_6c4d2afe3b74.slice/crio-5869db804a006d8b6032cc7c22b8e688ecf77155be14419b648fc7ca74300ae0 WatchSource:0}: Error finding container 5869db804a006d8b6032cc7c22b8e688ecf77155be14419b648fc7ca74300ae0: Status 404 returned error can't find the container with id 5869db804a006d8b6032cc7c22b8e688ecf77155be14419b648fc7ca74300ae0 Jul 17 12:22:33.593121 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.592564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvl9t" event={"ID":"61b4f3f4-920d-417f-9cf0-d3d60b64a434","Type":"ContainerStarted","Data":"fe38628a7b7f34c5bfd811f4546a4e32b4712fe235c904e845913269ed4c90bb"} Jul 17 12:22:33.594611 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:22:33.594583 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49468ee0_4570_43d8_b56b_ca2fb6ef2379.slice/crio-3a04b6066db17ea99dafa67e114598c435d417f7fa7b6c4ce6625a9c20e73fb6 WatchSource:0}: Error finding container 3a04b6066db17ea99dafa67e114598c435d417f7fa7b6c4ce6625a9c20e73fb6: Status 404 returned error can't find the container with id 3a04b6066db17ea99dafa67e114598c435d417f7fa7b6c4ce6625a9c20e73fb6 Jul 17 12:22:33.594712 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.594603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cp7zn" event={"ID":"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a","Type":"ContainerStarted","Data":"684a8e32f7ad166ba80d5f49d8af216fd1363e93bdb73a04b3eb9d93499808cb"} Jul 17 12:22:33.600946 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.600916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerStarted","Data":"91c20deddf53b4a5141320fb1edc382742d6d1cf49a1d6d6cd7f88245548ec2b"} Jul 17 12:22:33.602879 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:33.602847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-487k8" event={"ID":"b579fbbe-8aa3-476b-8895-6ad0daddd5a6","Type":"ContainerStarted","Data":"f6e0af85aa7ba0721da9b5e287aa633c2cbc89cc12d401fccb055db5b10b0009"} Jul 17 12:22:34.064379 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.064027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:34.064547 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.064484 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:34.064618 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.064563 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:36.064540772 +0000 UTC m=+6.009400258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:34.164969 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.164920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:34.165148 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.165000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:34.165148 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165141 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:34.165278 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165208 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:22:36.165187179 +0000 UTC m=+6.110046667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:34.165667 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165645 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:34.165749 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165677 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:34.165749 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165694 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:34.165749 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.165741 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:36.165724457 +0000 UTC m=+6.110583962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:34.582931 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.582894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:34.583678 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.583059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:34.583678 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.583489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:34.583678 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.583581 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:34.583678 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.583649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:34.583960 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:34.583722 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:34.622811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.622250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" event={"ID":"b9e412b5125944006d4709c364d63da6","Type":"ContainerStarted","Data":"ffad96d81a34e98384f1c397d1d7ef3be8e7c5d886c4b474f0532b224a449252"} Jul 17 12:22:34.639011 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.637844 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-112.ec2.internal" podStartSLOduration=3.637822815 podStartE2EDuration="3.637822815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:34.637553852 +0000 UTC m=+4.582413361" watchObservedRunningTime="2026-07-17 12:22:34.637822815 +0000 UTC m=+4.582682324" Jul 17 12:22:34.650349 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.650268 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2nfht" event={"ID":"5290d129-15a0-4b75-b59a-3878bf3e9e7b","Type":"ContainerStarted","Data":"8ab3a87ad5291c6a5e77804669e5b8ce2d3acc2b3f72f8bbd65e2f05582e6827"} Jul 17 12:22:34.657887 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.657846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz2nb" event={"ID":"f3f71dfb-6002-40d1-8004-9e57f9bcf384","Type":"ContainerStarted","Data":"4a423767b477512e2643bcce4dd001e8eeca05e7480ce46cebeb758346d8f989"} Jul 17 12:22:34.666972 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.666529 2568 generic.go:356] "Generic (PLEG): container finished" podID="f370020aead5ae2f513b709de0edb00e" containerID="192a33365913a0b00e0320fcaac469b6c4ff6d36b9c00b4a109451b143f43013" exitCode=0 Jul 17 12:22:34.666972 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.666649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" event={"ID":"f370020aead5ae2f513b709de0edb00e","Type":"ContainerDied","Data":"192a33365913a0b00e0320fcaac469b6c4ff6d36b9c00b4a109451b143f43013"} Jul 17 12:22:34.679458 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.679327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" event={"ID":"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf","Type":"ContainerStarted","Data":"1039bff79e1587a3380c9a3eaff14232e5b4d6f05ff2382dd90a27da3fd3344e"} Jul 17 12:22:34.695286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.695240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"3a04b6066db17ea99dafa67e114598c435d417f7fa7b6c4ce6625a9c20e73fb6"} Jul 17 12:22:34.704901 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:34.704861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" event={"ID":"0281067f-af52-408c-8faa-6c4d2afe3b74","Type":"ContainerStarted","Data":"5869db804a006d8b6032cc7c22b8e688ecf77155be14419b648fc7ca74300ae0"} Jul 17 12:22:35.721865 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:35.721829 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" event={"ID":"f370020aead5ae2f513b709de0edb00e","Type":"ContainerStarted","Data":"33652cce9bd1dd4b0af16383c3696fe8c75da1a3a380187ca79c2379faed10d4"} Jul 17 12:22:36.083505 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.083405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:36.083707 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.083635 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:36.083805 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.083720 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.083696965 +0000 UTC m=+10.028556451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:36.184382 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.184337 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:36.184565 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.184442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:36.184643 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.184599 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:36.184643 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.184620 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:36.184643 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.184633 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:36.184842 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.184696 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.184675996 +0000 UTC m=+10.129535483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:36.185180 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.185157 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:36.185253 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.185219 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.18520399 +0000 UTC m=+10.130063473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:36.583535 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.583438 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:36.583732 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.583598 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:36.584033 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.584012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:36.584130 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.584114 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:36.584193 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:36.584173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:36.584268 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:36.584248 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:38.581245 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:38.581214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:38.581735 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:38.581354 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:38.581815 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:38.581745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:38.581941 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:38.581858 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:38.581941 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:38.581900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:38.582111 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:38.581969 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:40.118120 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.118088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:40.118573 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.118180 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:40.118573 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.118247 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:48.118227139 +0000 UTC m=+18.063086622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.219199 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.219252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219406 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219465 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:22:48.219445979 +0000 UTC m=+18.164305463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219548 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219565 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219578 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:40.219989 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.219612 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:48.219601061 +0000 UTC m=+18.164460550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:40.581778 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.581673 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:40.581988 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.581817 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:40.581988 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.581834 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:40.581988 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:40.581951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:40.581988 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.581957 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:40.582190 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:40.582032 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:42.581805 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:42.581775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:42.581805 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:42.581790 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:42.582254 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:42.581909 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:42.582254 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:42.581944 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:42.582254 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:42.581991 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:42.582254 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:42.582108 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:44.581599 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:44.581561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:44.582052 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:44.581561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:44.582052 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:44.581563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:44.582052 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:44.581723 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:44.582052 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:44.581808 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:44.582052 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:44.581942 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:46.580884 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:46.580843 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:46.581313 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:46.580844 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:46.581313 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:46.580981 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:46.581313 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:46.580848 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:46.581313 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:46.581073 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:46.581313 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:46.581139 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:48.181418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.181375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:48.181977 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.181536 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:48.181977 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.181607 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:04.181587383 +0000 UTC m=+34.126446865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:22:48.281997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.281963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:48.281997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.282013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282129 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282137 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282160 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282170 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282202 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:23:04.282181792 +0000 UTC m=+34.227041278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:22:48.282250 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.282228 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:04.282209407 +0000 UTC m=+34.227068902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:22:48.581165 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.581076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:48.581333 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.581076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:48.581333 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.581210 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:48.581333 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:48.581225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:48.581333 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.581316 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:48.581504 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:48.581415 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:50.581764 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:50.581727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:50.582220 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:50.581827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:50.582220 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:50.581856 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:50.582220 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:50.581869 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:50.582220 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:50.581934 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:50.582220 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:50.582011 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:52.581152 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:52.581112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:52.581623 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:52.581113 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:52.581623 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:52.581249 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:52.581623 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:52.581340 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:52.581623 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:52.581112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:52.581623 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:52.581442 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:53.762863 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.762386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" event={"ID":"0281067f-af52-408c-8faa-6c4d2afe3b74","Type":"ContainerStarted","Data":"2ff8fad80e7a364fd20a7ff27241f02159ee14c9d0f905372427289090a8864f"} Jul 17 12:22:53.764287 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.764257 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cp7zn" event={"ID":"4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a","Type":"ContainerStarted","Data":"55170bf063f2a220791ffaa5b4a12a7a692d00f127d54dbd7ab9a694481392af"} Jul 17 12:22:53.766227 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.766199 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="234995d9a20cd9e5a230549513e44fd0875f6020301c9b8cf5ad6ea3037afed4" exitCode=0 Jul 17 12:22:53.766371 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.766286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"234995d9a20cd9e5a230549513e44fd0875f6020301c9b8cf5ad6ea3037afed4"} Jul 17 12:22:53.768070 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.768049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2nfht" event={"ID":"5290d129-15a0-4b75-b59a-3878bf3e9e7b","Type":"ContainerStarted","Data":"e75d0080c7c360be4a9d6ebef61ebda89a078d6a6b606ac90dce0257fe70b013"} Jul 17 12:22:53.769633 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.769606 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz2nb" event={"ID":"f3f71dfb-6002-40d1-8004-9e57f9bcf384","Type":"ContainerStarted","Data":"f9e9e8e0682959422ff3617b119fce18b35208314d7fd66966e3026199255b04"} Jul 17 12:22:53.771404 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.771377 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvl9t" event={"ID":"61b4f3f4-920d-417f-9cf0-d3d60b64a434","Type":"ContainerStarted","Data":"e89bbba75adbb8bb35b3647693db837e15a711e08dae7c029d3192ca39a072a1"} Jul 17 12:22:53.772890 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.772867 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" event={"ID":"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf","Type":"ContainerStarted","Data":"f46b6d3fde7b22573d186b0abcf81160026f0eb7b2a7ed2d7feb63f2cec00eed"} Jul 17 12:22:53.775957 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.775923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"756bb5b343a8fda143288a91251132d3c44cf9ed6a16aa7385a83dec7b015807"} Jul 17 12:22:53.776051 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.775987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"2202fd752fe619ff4eb59e0e5746f0635119edda7d811401c9943370f8b5dd69"} Jul 17 12:22:53.776051 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.776020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"9f1e2de78b9e9c4c28ad2df89f0baf2336466afa16049bf78ae1ae7e5185db23"} Jul 17 12:22:53.776132 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.776049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"0457f6c326aa0f161fa6ef952eabadf2af45d49369131b7f3c7079072659131d"} Jul 17 12:22:53.776132 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.776095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"d9e9853956fea66ae17c16729321e33a2e10e14eae918b76a8d4d088b8c63d54"} Jul 17 12:22:53.776132 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.776130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"742162114b97d8b8df07def115447940a193e348486b8e3577abab1a13281bd8"} Jul 17 12:22:53.777835 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.777784 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-112.ec2.internal" podStartSLOduration=22.777767003 podStartE2EDuration="22.777767003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:35.739882803 +0000 UTC m=+5.684742321" watchObservedRunningTime="2026-07-17 12:22:53.777767003 +0000 UTC m=+23.722626508" Jul 17 12:22:53.778421 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.778391 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tqwc" podStartSLOduration=3.563735076 podStartE2EDuration="22.778380197s" totalImagesPullingTime="19.214645121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:53.777551791 +0000 UTC m=+23.722411295" watchObservedRunningTime="2026-07-17 12:22:53.778380197 +0000 UTC m=+23.723239702" Jul 17 12:22:53.809161 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.809105 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvl9t" podStartSLOduration=3.616805544 podStartE2EDuration="22.809090174s" totalImagesPullingTime="19.19228463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:53.808849189 +0000 UTC m=+23.753708706" watchObservedRunningTime="2026-07-17 12:22:53.809090174 +0000 UTC m=+23.753949678" Jul 17 12:22:53.824088 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.824028 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sz2nb" podStartSLOduration=3.549452668 podStartE2EDuration="22.824009022s" totalImagesPullingTime="19.274556354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:53.82365342 +0000 UTC m=+23.768512924" watchObservedRunningTime="2026-07-17 12:22:53.824009022 +0000 UTC m=+23.768868528" Jul 17 12:22:53.836562 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:53.836515 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2nfht" podStartSLOduration=3.620024429 podStartE2EDuration="22.836499921s" totalImagesPullingTime="19.216475492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:53.83625852 +0000 UTC m=+23.781118023" watchObservedRunningTime="2026-07-17 12:22:53.836499921 +0000 UTC m=+23.781359422" Jul 17 12:22:54.210668 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.210627 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 12:22:54.513859 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.513705 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T12:22:54.210655223Z","UUID":"c11a1fe7-4b5a-422f-959b-91ce8ac080b8","Handler":null,"Name":"","Endpoint":""} Jul 17 12:22:54.515647 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.515405 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 12:22:54.515647 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.515454 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 12:22:54.581550 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.581512 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:54.581757 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.581627 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:54.581757 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:54.581632 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:54.581757 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.581654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:54.581936 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:54.581758 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:54.581936 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:54.581833 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:54.779786 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.779695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-487k8" event={"ID":"b579fbbe-8aa3-476b-8895-6ad0daddd5a6","Type":"ContainerStarted","Data":"f58cacb2fc598e2ef10fcf56dd493984d5b21f63e4cc3a6c4fe9be2258332615"} Jul 17 12:22:54.782637 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.782596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" event={"ID":"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf","Type":"ContainerStarted","Data":"450550a45d5e308c2da53bbf00a4d5aeb8bdc8bbe8492e6dbd314440c039fcb4"} Jul 17 12:22:54.793546 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.793489 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-487k8" podStartSLOduration=4.594815644 podStartE2EDuration="23.793468335s" totalImagesPullingTime="19.198652691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:54.793204107 +0000 UTC m=+24.738063610" watchObservedRunningTime="2026-07-17 12:22:54.793468335 +0000 UTC m=+24.738327830" Jul 17 12:22:54.794025 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.793992 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cp7zn" podStartSLOduration=8.306296249 podStartE2EDuration="23.793976281s" totalImagesPullingTime="15.487680032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:53.849156335 +0000 UTC m=+23.794015839" watchObservedRunningTime="2026-07-17 12:22:54.793976281 +0000 UTC m=+24.738835785" Jul 17 12:22:54.831501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.831467 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:54.832305 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:54.832278 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:22:55.786519 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:55.786485 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" event={"ID":"82c3e75d-af2b-4e7d-bac9-c4a3b89566cf","Type":"ContainerStarted","Data":"4fe9613348903836e7b443b8f7404fe3ad2815ed9922619caca46f6dfa0ebb28"} Jul 17 12:22:55.790090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:55.790056 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"844f302f6decb6fd1c96a7410fd60767682a13917c8529066a7c39493d461b66"} Jul 17 12:22:55.805266 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:55.805207 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8r8lb" podStartSLOduration=3.465425145 podStartE2EDuration="24.805185531s" totalImagesPullingTime="21.339760386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:55.804581312 +0000 UTC m=+25.749440818" watchObservedRunningTime="2026-07-17 12:22:55.805185531 +0000 UTC m=+25.750045036" Jul 17 12:22:56.581285 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:56.581244 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:56.581482 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:56.581244 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:56.581482 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:56.581375 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:56.581482 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:56.581466 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:56.581638 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:56.581254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:56.581638 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:56.581576 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:56.792418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:56.792387 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 12:22:58.581345 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:58.581166 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:22:58.581854 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:58.581166 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:22:58.581854 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:58.581443 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:22:58.581854 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:58.581166 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:22:58.581854 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:58.581495 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:22:58.581854 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:22:58.581627 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:22:59.801044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.801009 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="49e24e4822285f1c0e332c601fff4c364ca5454a0b7b75c6074c97c4e48a2967" exitCode=0 Jul 17 12:22:59.801903 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.801092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"49e24e4822285f1c0e332c601fff4c364ca5454a0b7b75c6074c97c4e48a2967"} Jul 17 12:22:59.804736 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.804713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" event={"ID":"49468ee0-4570-43d8-b56b-ca2fb6ef2379","Type":"ContainerStarted","Data":"9fbd4e3eb6ebc1bb95b129b24f9dd1daaa4567e6c6e0967172e58d4fb598338b"} Jul 17 12:22:59.805003 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.804984 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:59.805092 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.805059 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:59.820461 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.820434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:22:59.849080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:22:59.849030 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" podStartSLOduration=9.565916373 podStartE2EDuration="28.84901422s" totalImagesPullingTime="19.283097847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:59.848659065 +0000 UTC m=+29.793518570" watchObservedRunningTime="2026-07-17 12:22:59.84901422 +0000 UTC m=+29.793873722" Jul 17 12:23:00.582373 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:00.582178 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:00.582604 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:00.582226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:00.582604 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:00.582457 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:23:00.582604 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:00.582504 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:23:00.582604 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:00.582253 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:00.582604 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:00.582579 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:23:00.808039 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:00.808006 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:23:00.824885 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:00.824858 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:23:01.044150 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.044121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sp68r"] Jul 17 12:23:01.044315 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.044214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:01.044315 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:01.044291 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:23:01.047292 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.047265 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6t2j"] Jul 17 12:23:01.047428 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.047382 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:01.047527 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:01.047499 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:23:01.047941 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.047916 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2n479"] Jul 17 12:23:01.048041 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.047990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:01.048098 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:01.048064 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:23:01.691308 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.691272 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:23:01.691511 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.691424 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 12:23:01.691901 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.691882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cp7zn" Jul 17 12:23:01.811677 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.811648 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="23d34fc8be063931d3e45ec7767acc0976b9137961615e72f2de80ac971e2f4a" exitCode=0 Jul 17 12:23:01.812115 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:01.811724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"23d34fc8be063931d3e45ec7767acc0976b9137961615e72f2de80ac971e2f4a"} Jul 17 12:23:02.581453 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:02.581369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:02.581596 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:02.581379 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:02.581596 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:02.581484 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:23:02.581596 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:02.581571 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:23:02.581596 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:02.581379 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:02.581783 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:02.581728 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:23:03.818197 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:03.817953 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="67b554833d731997c941c88deb9e0668097507659ecd801bc406bbc0c14296d5" exitCode=0 Jul 17 12:23:03.818537 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:03.818041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"67b554833d731997c941c88deb9e0668097507659ecd801bc406bbc0c14296d5"} Jul 17 12:23:04.194861 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.194805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:04.195034 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.194965 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:23:04.195095 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.195038 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:36.195020514 +0000 UTC m=+66.139880001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 12:23:04.295794 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.295749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:04.295997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.295812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:04.295997 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.295952 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 12:23:04.295997 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.295961 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 12:23:04.295997 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.295979 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 12:23:04.295997 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.295989 2568 projected.go:196] Error preparing data for projected volume kube-api-access-z4kxl for pod openshift-network-diagnostics/network-check-target-sp68r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:23:04.296172 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.296030 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl podName:cffdd259-dc92-479d-acb5-4e5ba77a5b82 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:36.296016072 +0000 UTC m=+66.240875554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z4kxl" (UniqueName: "kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl") pod "network-check-target-sp68r" (UID: "cffdd259-dc92-479d-acb5-4e5ba77a5b82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 12:23:04.296172 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.296054 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret podName:f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db nodeName:}" failed. No retries permitted until 2026-07-17 12:23:36.296044144 +0000 UTC m=+66.240903630 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret") pod "global-pull-secret-syncer-2n479" (UID: "f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db") : object "kube-system"/"original-pull-secret" not registered Jul 17 12:23:04.580911 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.580823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:04.581080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.580826 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:04.581080 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.580957 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2n479" podUID="f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db" Jul 17 12:23:04.581080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:04.580974 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:04.581080 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.581060 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sp68r" podUID="cffdd259-dc92-479d-acb5-4e5ba77a5b82" Jul 17 12:23:04.581322 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:04.581164 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:23:05.338516 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.338492 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-112.ec2.internal" event="NodeReady" Jul 17 12:23:05.338843 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.338621 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 12:23:05.380230 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.380200 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n85zn"] Jul 17 12:23:05.384822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.384791 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rjdhb"] Jul 17 12:23:05.384975 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.384924 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.386454 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.386433 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.387153 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.387136 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 12:23:05.387153 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.387147 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 12:23:05.387309 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.387164 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\"" Jul 17 12:23:05.388571 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.388556 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 12:23:05.388781 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.388760 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btjhs\"" Jul 17 12:23:05.388781 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.388773 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 12:23:05.388936 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.388809 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 12:23:05.391878 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.391857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n85zn"] Jul 17 12:23:05.395066 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.395042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rjdhb"] Jul 17 12:23:05.506070 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdqt\" (UniqueName: \"kubernetes.io/projected/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-kube-api-access-6gdqt\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.506070 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxrv\" (UniqueName: \"kubernetes.io/projected/4f8dba38-59ce-4c23-8e17-5a713a852153-kube-api-access-hgxrv\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.506323 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-cert\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.506323 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506202 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dba38-59ce-4c23-8e17-5a713a852153-config-volume\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.506323 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.506323 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.506294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8dba38-59ce-4c23-8e17-5a713a852153-tmp-dir\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.606719 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdqt\" (UniqueName: \"kubernetes.io/projected/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-kube-api-access-6gdqt\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.606719 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxrv\" (UniqueName: \"kubernetes.io/projected/4f8dba38-59ce-4c23-8e17-5a713a852153-kube-api-access-hgxrv\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.606959 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-cert\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.606959 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dba38-59ce-4c23-8e17-5a713a852153-config-volume\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.606959 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.606959 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.606803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8dba38-59ce-4c23-8e17-5a713a852153-tmp-dir\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.606959 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:05.606927 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:05.607252 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:05.606987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:06.106966211 +0000 UTC m=+36.051825696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:05.607252 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.607179 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8dba38-59ce-4c23-8e17-5a713a852153-tmp-dir\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.607396 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.607376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dba38-59ce-4c23-8e17-5a713a852153-config-volume\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.612024 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.611994 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-cert\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.616503 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.616477 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxrv\" (UniqueName: \"kubernetes.io/projected/4f8dba38-59ce-4c23-8e17-5a713a852153-kube-api-access-hgxrv\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:05.622164 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.622132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdqt\" (UniqueName: \"kubernetes.io/projected/28ea3db6-f0ed-449c-9ad6-4e1253290b6d-kube-api-access-6gdqt\") pod \"ingress-canary-rjdhb\" (UID: \"28ea3db6-f0ed-449c-9ad6-4e1253290b6d\") " pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.702091 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.702038 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rjdhb" Jul 17 12:23:05.866382 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:05.866310 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rjdhb"] Jul 17 12:23:05.870880 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:23:05.870850 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ea3db6_f0ed_449c_9ad6_4e1253290b6d.slice/crio-47a6b10780861cdb213b60bfd687263df9620b1cb0ec30615b238a1962dc96aa WatchSource:0}: Error finding container 47a6b10780861cdb213b60bfd687263df9620b1cb0ec30615b238a1962dc96aa: Status 404 returned error can't find the container with id 47a6b10780861cdb213b60bfd687263df9620b1cb0ec30615b238a1962dc96aa Jul 17 12:23:06.111946 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.111896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:06.112202 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:06.112064 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:06.112202 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:06.112134 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:07.112118226 +0000 UTC m=+37.056977707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:06.581373 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.581338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:06.582228 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.581339 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:06.582228 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.581483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:06.585130 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585107 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 12:23:06.585467 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585446 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 12:23:06.585576 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 12:23:06.585576 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585539 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\"" Jul 17 12:23:06.585576 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585454 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 12:23:06.585751 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.585652 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\"" Jul 17 12:23:06.827666 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:06.827484 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rjdhb" event={"ID":"28ea3db6-f0ed-449c-9ad6-4e1253290b6d","Type":"ContainerStarted","Data":"47a6b10780861cdb213b60bfd687263df9620b1cb0ec30615b238a1962dc96aa"} Jul 17 12:23:07.121538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:07.121498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:07.121749 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:07.121646 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:07.121749 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:07.121737 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:09.121714898 +0000 UTC m=+39.066574395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:09.135823 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:09.135786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:09.136265 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:09.135975 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:09.136265 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:09.136058 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:13.136035823 +0000 UTC m=+43.080895327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:09.838627 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:09.838545 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="d57b98f25e3f45c431aac4441ea285225a91c9dc815fef4e0e8d8323e44b6050" exitCode=0 Jul 17 12:23:09.838790 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:09.838616 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"d57b98f25e3f45c431aac4441ea285225a91c9dc815fef4e0e8d8323e44b6050"} Jul 17 12:23:09.840220 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:09.840194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rjdhb" event={"ID":"28ea3db6-f0ed-449c-9ad6-4e1253290b6d","Type":"ContainerStarted","Data":"386e53c09ffedbc79ea3d322a4a71c46b2d1cc821d3a88a6f3e5e94c5b890f35"} Jul 17 12:23:09.872562 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:09.872490 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rjdhb" podStartSLOduration=1.373441121 podStartE2EDuration="4.872465238s" totalImagesPullingTime="3.499024117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:09.871907537 +0000 UTC m=+39.816767043" watchObservedRunningTime="2026-07-17 12:23:09.872465238 +0000 UTC m=+39.817324743" Jul 17 12:23:10.845371 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:10.845336 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0" containerID="5fe1bffd295c9015705f7a8b166d6023fb32066ab5df9d1e27223022213409ea" exitCode=0 Jul 17 12:23:10.845754 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:10.845413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerDied","Data":"5fe1bffd295c9015705f7a8b166d6023fb32066ab5df9d1e27223022213409ea"} Jul 17 12:23:11.851018 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:11.850973 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" event={"ID":"ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0","Type":"ContainerStarted","Data":"b9684da8dbecc6e823af8c22a79131f274813335bf9aa73e1412c18d14b848a8"} Jul 17 12:23:11.871377 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:11.871323 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6v7ml" podStartSLOduration=9.71145028 podStartE2EDuration="41.871303384s" totalImagesPullingTime="32.159853104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:11.87026725 +0000 UTC m=+41.815126765" watchObservedRunningTime="2026-07-17 12:23:11.871303384 +0000 UTC m=+41.816162888" Jul 17 12:23:13.165023 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:13.164982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:13.165381 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:13.165140 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:13.165381 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:13.165201 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:21.16518535 +0000 UTC m=+51.110044835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:21.219624 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:21.219574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:21.220231 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:21.219736 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:21.220231 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:21.219832 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:37.219808743 +0000 UTC m=+67.164668232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:32.824271 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:32.824244 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-png4r" Jul 17 12:23:36.221634 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.221585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:23:36.224128 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.224109 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 12:23:36.232818 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:36.232797 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 12:23:36.232922 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:36.232910 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:24:40.232891014 +0000 UTC m=+130.177750682 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : secret "metrics-daemon-secret" not found Jul 17 12:23:36.321928 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.321879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:36.321928 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.321936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:36.324477 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.324460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 12:23:36.324543 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.324510 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 12:23:36.335079 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.335055 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 12:23:36.336971 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.336951 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db-original-pull-secret\") pod \"global-pull-secret-syncer-2n479\" (UID: \"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db\") " pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:36.346310 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.346283 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kxl\" (UniqueName: \"kubernetes.io/projected/cffdd259-dc92-479d-acb5-4e5ba77a5b82-kube-api-access-z4kxl\") pod \"network-check-target-sp68r\" (UID: \"cffdd259-dc92-479d-acb5-4e5ba77a5b82\") " pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:36.594458 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.594378 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2n479" Jul 17 12:23:36.605141 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.605118 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\"" Jul 17 12:23:36.613114 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.613090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:36.728025 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.727992 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2n479"] Jul 17 12:23:36.731093 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:23:36.731061 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f9c9d3_e58c_4eb0_8319_ded8fdf4a4db.slice/crio-9d3c2644a97e23138cde8a24498140f2efeace90f4110def6dfd31c316e57f64 WatchSource:0}: Error finding container 9d3c2644a97e23138cde8a24498140f2efeace90f4110def6dfd31c316e57f64: Status 404 returned error can't find the container with id 9d3c2644a97e23138cde8a24498140f2efeace90f4110def6dfd31c316e57f64 Jul 17 12:23:36.745584 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.745559 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sp68r"] Jul 17 12:23:36.748404 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:23:36.748382 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffdd259_dc92_479d_acb5_4e5ba77a5b82.slice/crio-3ea422f83f1c47e8e3168d72437a0a4e065d08dbd74f382343c0de0c6f782bf0 WatchSource:0}: Error finding container 3ea422f83f1c47e8e3168d72437a0a4e065d08dbd74f382343c0de0c6f782bf0: Status 404 returned error can't find the container with id 3ea422f83f1c47e8e3168d72437a0a4e065d08dbd74f382343c0de0c6f782bf0 Jul 17 12:23:36.908520 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.908483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sp68r" event={"ID":"cffdd259-dc92-479d-acb5-4e5ba77a5b82","Type":"ContainerStarted","Data":"3ea422f83f1c47e8e3168d72437a0a4e065d08dbd74f382343c0de0c6f782bf0"} Jul 17 12:23:36.909550 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:36.909529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2n479" event={"ID":"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db","Type":"ContainerStarted","Data":"9d3c2644a97e23138cde8a24498140f2efeace90f4110def6dfd31c316e57f64"} Jul 17 12:23:37.230043 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:37.229415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:23:37.230043 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:37.229583 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:23:37.230043 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:23:37.229649 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:24:09.229633486 +0000 UTC m=+99.174492970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:23:42.929222 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:42.929185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sp68r" event={"ID":"cffdd259-dc92-479d-acb5-4e5ba77a5b82","Type":"ContainerStarted","Data":"9f12cb7f5e9a39f3dcec81ee651c9da187d7ed97a8feb8e5d2d7a81af59721a6"} Jul 17 12:23:42.929701 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:42.929262 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:23:42.930597 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:42.930578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2n479" event={"ID":"f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db","Type":"ContainerStarted","Data":"c1191b95eefb5569882f67c8fa0cc150ef7794675dc6f27ccbfb3dc6d158cdb0"} Jul 17 12:23:42.944704 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:42.944642 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sp68r" podStartSLOduration=67.691120261 podStartE2EDuration="1m12.944627715s" totalImagesPullingTime="5.253507454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:42.944335955 +0000 UTC m=+72.889195457" watchObservedRunningTime="2026-07-17 12:23:42.944627715 +0000 UTC m=+72.889487220" Jul 17 12:23:42.957182 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:23:42.957134 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2n479" podStartSLOduration=66.681806853 podStartE2EDuration="1m11.957117873s" totalImagesPullingTime="5.27531102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:42.957025795 +0000 UTC m=+72.901885298" watchObservedRunningTime="2026-07-17 12:23:42.957117873 +0000 UTC m=+72.901977358" Jul 17 12:24:09.250303 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:24:09.250183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:24:09.250762 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:24:09.250306 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:24:09.250762 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:24:09.250377 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:25:13.250359915 +0000 UTC m=+163.195219402 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:24:13.936024 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:24:13.935992 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sp68r" Jul 17 12:24:40.264044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:24:40.263985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:24:40.264543 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:24:40.264165 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 12:24:40.264543 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:24:40.264250 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.264231565 +0000 UTC m=+252.209091048 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : secret "metrics-daemon-secret" not found Jul 17 12:25:08.397785 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:25:08.397745 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n85zn" podUID="4f8dba38-59ce-4c23-8e17-5a713a852153" Jul 17 12:25:09.137084 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:25:09.137057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n85zn" Jul 17 12:25:09.610154 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:25:09.610105 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:25:13.289166 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:25:13.289125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:25:13.289554 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:25:13.289237 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 12:25:13.289554 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:25:13.289297 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls podName:4f8dba38-59ce-4c23-8e17-5a713a852153 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:15.28928338 +0000 UTC m=+285.234142862 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls") pod "dns-default-n85zn" (UID: "4f8dba38-59ce-4c23-8e17-5a713a852153") : secret "dns-default-metrics-tls" not found Jul 17 12:25:23.581387 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:25:23.581289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:26:26.967071 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.967032 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn"] Jul 17 12:26:26.969084 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.969066 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k"] Jul 17 12:26:26.969196 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.969177 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:26.970986 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.970967 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cjcz9"] Jul 17 12:26:26.971113 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.971077 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" Jul 17 12:26:26.974584 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.974561 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 17 12:26:26.974795 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.974569 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 12:26:26.975879 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.975856 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bb9845cfd-95zm4"] Jul 17 12:26:26.975879 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.975867 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 12:26:26.976049 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.975940 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:26.976112 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.976058 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 17 12:26:26.976173 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.976160 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 17 12:26:26.976437 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.976395 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dpnsj\"" Jul 17 12:26:26.976564 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.976460 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 17 12:26:26.976564 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.976466 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6d84b\"" Jul 17 12:26:26.978200 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978181 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5lfms\"" Jul 17 12:26:26.978289 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978181 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 17 12:26:26.978478 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978462 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 17 12:26:26.978521 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978467 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 12:26:26.978766 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978750 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:26.978929 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.978909 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 12:26:26.983850 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.983824 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 17 12:26:26.984167 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 17 12:26:26.984272 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984254 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 12:26:26.984405 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984391 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 12:26:26.984476 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984453 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 17 12:26:26.984731 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984711 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g2jrx\"" Jul 17 12:26:26.984828 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984739 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 17 12:26:26.984828 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.984788 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k"] Jul 17 12:26:26.986715 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.986674 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 17 12:26:26.987481 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.987452 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn"] Jul 17 12:26:26.988416 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.988369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cjcz9"] Jul 17 12:26:26.989835 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:26.989816 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb9845cfd-95zm4"] Jul 17 12:26:27.054153 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.054117 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8"] Jul 17 12:26:27.056172 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.056154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.058622 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.058601 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5qbh6\"" Jul 17 12:26:27.058751 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.058660 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 17 12:26:27.058751 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.058741 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 17 12:26:27.058858 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.058751 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 17 12:26:27.058858 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.058846 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 17 12:26:27.070261 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.070228 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8"] Jul 17 12:26:27.074593 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-stats-auth\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.074746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdwj\" (UniqueName: \"kubernetes.io/projected/4a85a944-52d1-4a4b-a48f-08967187bf33-kube-api-access-tqdwj\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.074746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-default-certificate\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.074746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.074746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.074746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67gg\" (UniqueName: \"kubernetes.io/projected/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-kube-api-access-j67gg\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074756 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-tmp\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074796 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074818 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fft\" (UniqueName: \"kubernetes.io/projected/6468c778-c914-484e-83a9-eae7fe6f52d0-kube-api-access-f8fft\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjk4x\" (UniqueName: \"kubernetes.io/projected/0fa0e48a-d489-44d1-a3e8-30d99a0b0066-kube-api-access-gjk4x\") pod \"volume-data-source-validator-78cf4c6f6-9m95k\" (UID: \"0fa0e48a-d489-44d1-a3e8-30d99a0b0066\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" Jul 17 12:26:27.074938 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-snapshots\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.075293 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.075293 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.074978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6468c778-c914-484e-83a9-eae7fe6f52d0-serving-cert\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.175542 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-stats-auth\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.175542 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdwj\" (UniqueName: \"kubernetes.io/projected/4a85a944-52d1-4a4b-a48f-08967187bf33-kube-api-access-tqdwj\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-default-certificate\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175645 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.175757 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 12:26:27.175802 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.175799 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.675776577 +0000 UTC m=+237.620636086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : configmap references non-existent config key: service-ca.crt Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67gg\" (UniqueName: \"kubernetes.io/projected/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-kube-api-access-j67gg\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-tmp\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.175884 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.675867942 +0000 UTC m=+237.620727432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : secret "router-metrics-certs-default" not found Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.175994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxc4\" (UniqueName: \"kubernetes.io/projected/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-kube-api-access-jrxc4\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176101 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fft\" (UniqueName: \"kubernetes.io/projected/6468c778-c914-484e-83a9-eae7fe6f52d0-kube-api-access-f8fft\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjk4x\" (UniqueName: \"kubernetes.io/projected/0fa0e48a-d489-44d1-a3e8-30d99a0b0066-kube-api-access-gjk4x\") pod \"volume-data-source-validator-78cf4c6f6-9m95k\" (UID: \"0fa0e48a-d489-44d1-a3e8-30d99a0b0066\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" Jul 17 12:26:27.176501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-snapshots\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.176501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6468c778-c914-484e-83a9-eae7fe6f52d0-serving-cert\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.176767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.176570 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:27.176767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.176621 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.676603857 +0000 UTC m=+237.621463365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:27.176767 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176767 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-tmp\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.176954 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.176834 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6468c778-c914-484e-83a9-eae7fe6f52d0-snapshots\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.177181 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.177158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.177260 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.177221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468c778-c914-484e-83a9-eae7fe6f52d0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.178864 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.178845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6468c778-c914-484e-83a9-eae7fe6f52d0-serving-cert\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.179074 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.179057 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-stats-auth\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.179123 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.179114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-default-certificate\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.190818 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.190796 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67gg\" (UniqueName: \"kubernetes.io/projected/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-kube-api-access-j67gg\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.193841 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.193793 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fft\" (UniqueName: \"kubernetes.io/projected/6468c778-c914-484e-83a9-eae7fe6f52d0-kube-api-access-f8fft\") pod \"insights-operator-555d9d7885-cjcz9\" (UID: \"6468c778-c914-484e-83a9-eae7fe6f52d0\") " pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.194701 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.194658 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdwj\" (UniqueName: \"kubernetes.io/projected/4a85a944-52d1-4a4b-a48f-08967187bf33-kube-api-access-tqdwj\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.194792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.194701 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjk4x\" (UniqueName: \"kubernetes.io/projected/0fa0e48a-d489-44d1-a3e8-30d99a0b0066-kube-api-access-gjk4x\") pod \"volume-data-source-validator-78cf4c6f6-9m95k\" (UID: \"0fa0e48a-d489-44d1-a3e8-30d99a0b0066\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" Jul 17 12:26:27.277235 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.277144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.277235 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.277196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.277405 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.277239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxc4\" (UniqueName: \"kubernetes.io/projected/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-kube-api-access-jrxc4\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.277847 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.277826 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.279607 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.279579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.285395 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.285374 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxc4\" (UniqueName: \"kubernetes.io/projected/37431f8a-bcda-4b4d-b8a2-6dc14a402a5f-kube-api-access-jrxc4\") pod \"kube-storage-version-migrator-operator-7658d9b749-kl8f8\" (UID: \"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.290386 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.290373 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" Jul 17 12:26:27.296808 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.296790 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" Jul 17 12:26:27.364631 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.364601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" Jul 17 12:26:27.417697 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.417642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k"] Jul 17 12:26:27.425599 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:27.425569 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa0e48a_d489_44d1_a3e8_30d99a0b0066.slice/crio-a59fa275f524a8d1075f55903f7b9b3d7abe9b9db671914839bed94c913d1366 WatchSource:0}: Error finding container a59fa275f524a8d1075f55903f7b9b3d7abe9b9db671914839bed94c913d1366: Status 404 returned error can't find the container with id a59fa275f524a8d1075f55903f7b9b3d7abe9b9db671914839bed94c913d1366 Jul 17 12:26:27.445119 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.445092 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cjcz9"] Jul 17 12:26:27.449210 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:27.449180 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468c778_c914_484e_83a9_eae7fe6f52d0.slice/crio-3c5c0ceedae4794e6b73b4a0b91b4b17eebe7c26961bebe2c63ae980feacd1b4 WatchSource:0}: Error finding container 3c5c0ceedae4794e6b73b4a0b91b4b17eebe7c26961bebe2c63ae980feacd1b4: Status 404 returned error can't find the container with id 3c5c0ceedae4794e6b73b4a0b91b4b17eebe7c26961bebe2c63ae980feacd1b4 Jul 17 12:26:27.497355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.497326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8"] Jul 17 12:26:27.500467 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:27.500443 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37431f8a_bcda_4b4d_b8a2_6dc14a402a5f.slice/crio-46fdac1e9a0aa2c50289e3589a7a5fb2709b7031d9bc7b6b724901dfdbd72aa8 WatchSource:0}: Error finding container 46fdac1e9a0aa2c50289e3589a7a5fb2709b7031d9bc7b6b724901dfdbd72aa8: Status 404 returned error can't find the container with id 46fdac1e9a0aa2c50289e3589a7a5fb2709b7031d9bc7b6b724901dfdbd72aa8 Jul 17 12:26:27.679789 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.679760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:27.679939 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.679819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.679939 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:27.679837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:27.679939 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.679900 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:27.680032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.679939 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.679920526 +0000 UTC m=+238.624780010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : configmap references non-existent config key: service-ca.crt Jul 17 12:26:27.680032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.679952 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 12:26:27.680032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.679959 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.679951847 +0000 UTC m=+238.624811329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:27.680032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:27.679989 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.679975652 +0000 UTC m=+238.624835134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : secret "router-metrics-certs-default" not found Jul 17 12:26:28.320893 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.320844 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" event={"ID":"6468c778-c914-484e-83a9-eae7fe6f52d0","Type":"ContainerStarted","Data":"3c5c0ceedae4794e6b73b4a0b91b4b17eebe7c26961bebe2c63ae980feacd1b4"} Jul 17 12:26:28.322338 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.322270 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" event={"ID":"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f","Type":"ContainerStarted","Data":"46fdac1e9a0aa2c50289e3589a7a5fb2709b7031d9bc7b6b724901dfdbd72aa8"} Jul 17 12:26:28.323800 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.323765 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" event={"ID":"0fa0e48a-d489-44d1-a3e8-30d99a0b0066","Type":"ContainerStarted","Data":"a59fa275f524a8d1075f55903f7b9b3d7abe9b9db671914839bed94c913d1366"} Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.689090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.689194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:28.689224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:28.689263 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:28.689341 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:28.689350 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.6893262 +0000 UTC m=+240.634185717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:28.689378 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.689358527 +0000 UTC m=+240.634218009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : configmap references non-existent config key: service-ca.crt Jul 17 12:26:28.689406 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:28.689414 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.68940324 +0000 UTC m=+240.634262726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : secret "router-metrics-certs-default" not found Jul 17 12:26:29.328343 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:29.328308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" event={"ID":"0fa0e48a-d489-44d1-a3e8-30d99a0b0066","Type":"ContainerStarted","Data":"13a4a589d5984e800cbbf4daa5859489323e11c66bfdedc1be9397d97b3c04ac"} Jul 17 12:26:29.343283 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:29.343233 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-9m95k" podStartSLOduration=1.89769026 podStartE2EDuration="3.343212296s" totalImagesPullingTime="1.445522036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:29.342447437 +0000 UTC m=+239.287306941" watchObservedRunningTime="2026-07-17 12:26:29.343212296 +0000 UTC m=+239.288071801" Jul 17 12:26:30.333354 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.333259 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" event={"ID":"6468c778-c914-484e-83a9-eae7fe6f52d0","Type":"ContainerStarted","Data":"62a4237389c845c93910c8580bc015db8d9b49b7724c829009616bb6028d53ec"} Jul 17 12:26:30.334822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.334796 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" event={"ID":"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f","Type":"ContainerStarted","Data":"d8ed1a3e4df52a30bfa67e8f2c47c09f745db14da0c2b0ccf896d2a31dfc3646"} Jul 17 12:26:30.348822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.348770 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" podStartSLOduration=1.8834345190000001 podStartE2EDuration="4.348751185s" totalImagesPullingTime="2.465316666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:30.348648799 +0000 UTC m=+240.293508330" watchObservedRunningTime="2026-07-17 12:26:30.348751185 +0000 UTC m=+240.293610689" Jul 17 12:26:30.363301 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.363252 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" podStartSLOduration=0.946330042 podStartE2EDuration="3.363237489s" totalImagesPullingTime="2.416907447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:30.362241603 +0000 UTC m=+240.307101107" watchObservedRunningTime="2026-07-17 12:26:30.363237489 +0000 UTC m=+240.308096989" Jul 17 12:26:30.705626 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.705582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:30.705821 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.705672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:30.705821 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.705718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:30.705821 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:30.705757 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:30.705821 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:30.705813 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.705789551 +0000 UTC m=+244.650649053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : configmap references non-existent config key: service-ca.crt Jul 17 12:26:30.706025 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:30.705840 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.705831039 +0000 UTC m=+244.650690537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:30.706025 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:30.705865 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 12:26:30.706025 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:30.705915 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.705899345 +0000 UTC m=+244.650758830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : secret "router-metrics-certs-default" not found Jul 17 12:26:30.785118 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.785087 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h27zj"] Jul 17 12:26:30.787364 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.787349 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:30.790847 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.790826 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmxgd\"" Jul 17 12:26:30.791070 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.791042 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 12:26:30.791177 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.791048 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 12:26:30.817757 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.817662 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h27zj"] Jul 17 12:26:30.907380 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.907344 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7793de-4484-4e91-8ea9-6b32478a70fd-data-volume\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:30.907380 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.907383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:30.907599 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.907418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7793de-4484-4e91-8ea9-6b32478a70fd-crio-socket\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:30.907599 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.907509 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:30.907599 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:30.907565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9wq\" (UniqueName: \"kubernetes.io/projected/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-api-access-th9wq\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008726 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7793de-4484-4e91-8ea9-6b32478a70fd-crio-socket\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008726 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9wq\" (UniqueName: \"kubernetes.io/projected/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-api-access-th9wq\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7793de-4484-4e91-8ea9-6b32478a70fd-crio-socket\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7793de-4484-4e91-8ea9-6b32478a70fd-data-volume\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.008828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.008888 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 12:26:31.008918 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.008898 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 12:26:31.009092 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.008959 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.508941314 +0000 UTC m=+241.453800798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : configmap "kube-rbac-proxy" not found Jul 17 12:26:31.009092 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.008976 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.508967831 +0000 UTC m=+241.453827319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : secret "insights-runtime-extractor-tls" not found Jul 17 12:26:31.009165 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.009135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7793de-4484-4e91-8ea9-6b32478a70fd-data-volume\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.016941 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.016908 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9wq\" (UniqueName: \"kubernetes.io/projected/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-api-access-th9wq\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.513274 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.513241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.513767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.513342 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 12:26:31.513767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.513402 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.513382461 +0000 UTC m=+242.458241944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : secret "insights-runtime-extractor-tls" not found Jul 17 12:26:31.513767 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:31.513495 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:31.513767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.513601 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 12:26:31.513767 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:31.513671 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.513653599 +0000 UTC m=+242.458513084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : configmap "kube-rbac-proxy" not found Jul 17 12:26:32.522005 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:32.521946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:32.522454 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:32.522071 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:32.522454 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:32.522102 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 12:26:32.522454 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:32.522165 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.522150303 +0000 UTC m=+244.467009784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : secret "insights-runtime-extractor-tls" not found Jul 17 12:26:32.522454 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:32.522180 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 12:26:32.522454 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:32.522228 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.522215523 +0000 UTC m=+244.467075005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : configmap "kube-rbac-proxy" not found Jul 17 12:26:32.559754 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:32.559726 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2nfht_5290d129-15a0-4b75-b59a-3878bf3e9e7b/dns-node-resolver/0.log" Jul 17 12:26:33.761253 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:33.761225 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvl9t_61b4f3f4-920d-417f-9cf0-d3d60b64a434/node-ca/0.log" Jul 17 12:26:34.536845 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.536814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:34.537008 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.536873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:34.537008 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.536954 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 12:26:34.537008 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.536977 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 12:26:34.537129 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.537021 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.537000993 +0000 UTC m=+248.481860496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : configmap "kube-rbac-proxy" not found Jul 17 12:26:34.537129 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.537035 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.537029316 +0000 UTC m=+248.481888799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : secret "insights-runtime-extractor-tls" not found Jul 17 12:26:34.738059 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.738018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:34.738205 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.738080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:34.738205 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.738134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:34.738205 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.738158 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:34.738340 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.738219 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.738203634 +0000 UTC m=+252.683063116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:34.738340 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.738243 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.738227377 +0000 UTC m=+252.683086859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : configmap references non-existent config key: service-ca.crt Jul 17 12:26:34.738340 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.738243 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 12:26:34.738340 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:34.738290 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs podName:4a85a944-52d1-4a4b-a48f-08967187bf33 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.738278462 +0000 UTC m=+252.683137946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs") pod "router-default-5bb9845cfd-95zm4" (UID: "4a85a944-52d1-4a4b-a48f-08967187bf33") : secret "router-metrics-certs-default" not found Jul 17 12:26:34.760749 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:34.760714 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rjdhb_28ea3db6-f0ed-449c-9ad6-4e1253290b6d/serve-healthcheck-canary/0.log" Jul 17 12:26:38.567620 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:38.567582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:38.568032 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:38.567663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:38.568032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:38.567737 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 12:26:38.568032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:38.567774 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 12:26:38.568032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:38.567806 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.567788032 +0000 UTC m=+256.512647514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : configmap "kube-rbac-proxy" not found Jul 17 12:26:38.568032 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:38.567824 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls podName:ab7793de-4484-4e91-8ea9-6b32478a70fd nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.56781256 +0000 UTC m=+256.512672042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27zj" (UID: "ab7793de-4484-4e91-8ea9-6b32478a70fd") : secret "insights-runtime-extractor-tls" not found Jul 17 12:26:41.976736 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.976677 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6756c7dc49-pw49x"] Jul 17 12:26:41.978920 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.978900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.981400 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.981375 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 12:26:41.981530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.981375 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5tvcb\"" Jul 17 12:26:41.981530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.981375 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 12:26:41.981530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.981422 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 12:26:41.985723 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.985583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 12:26:41.991611 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.991571 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6756c7dc49-pw49x"] Jul 17 12:26:41.994875 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.994849 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-image-registry-private-configuration\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.994997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.994889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzcj\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-kube-api-access-jlzcj\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.994997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.994917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-tls\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.994997 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.994939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-certificates\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.995174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.994996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-ca-trust-extracted\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.995174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.995035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-trusted-ca\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.995174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.995138 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-bound-sa-token\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:41.995174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:41.995168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-installation-pull-secrets\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096127 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-bound-sa-token\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096127 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-installation-pull-secrets\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-image-registry-private-configuration\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzcj\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-kube-api-access-jlzcj\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-tls\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-certificates\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-ca-trust-extracted\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096575 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-trusted-ca\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.096908 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.096882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-ca-trust-extracted\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.097200 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.097185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-certificates\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.097798 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.097775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-trusted-ca\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.098945 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.098915 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-image-registry-private-configuration\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.099034 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.098972 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-registry-tls\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.099034 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.098970 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-installation-pull-secrets\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.103418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.103391 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-bound-sa-token\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.103620 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.103603 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzcj\" (UniqueName: \"kubernetes.io/projected/12a52a13-07ec-4c12-bf1d-698dc50ae6a2-kube-api-access-jlzcj\") pod \"image-registry-6756c7dc49-pw49x\" (UID: \"12a52a13-07ec-4c12-bf1d-698dc50ae6a2\") " pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.288060 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.287970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:42.299048 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.299011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:26:42.299273 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:42.299248 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 12:26:42.299385 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:42.299328 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs podName:3ae54f7a-49e7-4701-a8e8-af4f183d4d09 nodeName:}" failed. No retries permitted until 2026-07-17 12:28:44.29930533 +0000 UTC m=+374.244164812 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs") pod "network-metrics-daemon-l6t2j" (UID: "3ae54f7a-49e7-4701-a8e8-af4f183d4d09") : secret "metrics-daemon-secret" not found Jul 17 12:26:42.418032 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.417998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6756c7dc49-pw49x"] Jul 17 12:26:42.421224 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:42.421195 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a52a13_07ec_4c12_bf1d_698dc50ae6a2.slice/crio-0129e0525674800ff97bd9f3ffff598c9dcf4946fbb110c6eac200c77d1b1216 WatchSource:0}: Error finding container 0129e0525674800ff97bd9f3ffff598c9dcf4946fbb110c6eac200c77d1b1216: Status 404 returned error can't find the container with id 0129e0525674800ff97bd9f3ffff598c9dcf4946fbb110c6eac200c77d1b1216 Jul 17 12:26:42.802226 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.802194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:42.802389 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.802250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:42.802389 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.802291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:42.802389 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:42.802355 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:42.802503 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:26:42.802425 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls podName:2c95494b-39b9-4e8e-bcf1-8d722cd56a6f nodeName:}" failed. No retries permitted until 2026-07-17 12:26:58.802409744 +0000 UTC m=+268.747269230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7v5bn" (UID: "2c95494b-39b9-4e8e-bcf1-8d722cd56a6f") : secret "cluster-monitoring-operator-tls" not found Jul 17 12:26:42.803159 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.803121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a85a944-52d1-4a4b-a48f-08967187bf33-service-ca-bundle\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:42.804708 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.804673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a85a944-52d1-4a4b-a48f-08967187bf33-metrics-certs\") pod \"router-default-5bb9845cfd-95zm4\" (UID: \"4a85a944-52d1-4a4b-a48f-08967187bf33\") " pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:42.904325 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.904297 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-g2jrx\"" Jul 17 12:26:42.913117 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:42.913088 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:43.036337 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.036306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bb9845cfd-95zm4"] Jul 17 12:26:43.039139 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:43.039103 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a85a944_52d1_4a4b_a48f_08967187bf33.slice/crio-288b83560f673f82e8d33424748bded6487a339447149795b568cb9b6c5a7d23 WatchSource:0}: Error finding container 288b83560f673f82e8d33424748bded6487a339447149795b568cb9b6c5a7d23: Status 404 returned error can't find the container with id 288b83560f673f82e8d33424748bded6487a339447149795b568cb9b6c5a7d23 Jul 17 12:26:43.376891 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.376796 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" event={"ID":"4a85a944-52d1-4a4b-a48f-08967187bf33","Type":"ContainerStarted","Data":"1066da4a2dc76657d092715bce81eba7ad5077079d081e05ca520ec90a062a27"} Jul 17 12:26:43.376891 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.376845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" event={"ID":"4a85a944-52d1-4a4b-a48f-08967187bf33","Type":"ContainerStarted","Data":"288b83560f673f82e8d33424748bded6487a339447149795b568cb9b6c5a7d23"} Jul 17 12:26:43.378230 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.378207 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" event={"ID":"12a52a13-07ec-4c12-bf1d-698dc50ae6a2","Type":"ContainerStarted","Data":"01c21da2619159cbbca1b243d214ad19630822189950db8c7799c9c3363fd2fe"} Jul 17 12:26:43.378338 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.378242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" event={"ID":"12a52a13-07ec-4c12-bf1d-698dc50ae6a2","Type":"ContainerStarted","Data":"0129e0525674800ff97bd9f3ffff598c9dcf4946fbb110c6eac200c77d1b1216"} Jul 17 12:26:43.378374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.378344 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:26:43.394110 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.394065 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" podStartSLOduration=17.394050388 podStartE2EDuration="17.394050388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:43.393005196 +0000 UTC m=+253.337864699" watchObservedRunningTime="2026-07-17 12:26:43.394050388 +0000 UTC m=+253.338909891" Jul 17 12:26:43.411931 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.411885 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" podStartSLOduration=2.411869752 podStartE2EDuration="2.411869752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:43.41087638 +0000 UTC m=+253.355735895" watchObservedRunningTime="2026-07-17 12:26:43.411869752 +0000 UTC m=+253.356729262" Jul 17 12:26:43.913751 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.913719 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:43.916138 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:43.916119 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:44.381649 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:44.381559 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:44.382895 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:44.382874 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bb9845cfd-95zm4" Jul 17 12:26:46.637572 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.637522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:46.637998 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.637599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:46.638193 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.638170 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7793de-4484-4e91-8ea9-6b32478a70fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:46.639994 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.639979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7793de-4484-4e91-8ea9-6b32478a70fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27zj\" (UID: \"ab7793de-4484-4e91-8ea9-6b32478a70fd\") " pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:46.697378 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.697326 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h27zj" Jul 17 12:26:46.822235 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:46.822200 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h27zj"] Jul 17 12:26:46.825500 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:46.825470 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7793de_4484_4e91_8ea9_6b32478a70fd.slice/crio-2564489b1d087b8c1964d42d45484168f140b27a5c494e4d9748aa8651667538 WatchSource:0}: Error finding container 2564489b1d087b8c1964d42d45484168f140b27a5c494e4d9748aa8651667538: Status 404 returned error can't find the container with id 2564489b1d087b8c1964d42d45484168f140b27a5c494e4d9748aa8651667538 Jul 17 12:26:47.392960 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:47.392922 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27zj" event={"ID":"ab7793de-4484-4e91-8ea9-6b32478a70fd","Type":"ContainerStarted","Data":"2564489b1d087b8c1964d42d45484168f140b27a5c494e4d9748aa8651667538"} Jul 17 12:26:52.414261 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:52.414182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27zj" event={"ID":"ab7793de-4484-4e91-8ea9-6b32478a70fd","Type":"ContainerStarted","Data":"552471ecf2c8662a2ab033ba672edc7541a1d1a0cd72a72f9504204e43abec1f"} Jul 17 12:26:55.426871 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:55.426834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27zj" event={"ID":"ab7793de-4484-4e91-8ea9-6b32478a70fd","Type":"ContainerStarted","Data":"3a679796aa17adc93e31bef3da0828319ff1ed80ca4c7dd708356675bb768a01"} Jul 17 12:26:58.841565 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:58.841468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:58.843981 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:58.843960 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c95494b-39b9-4e8e-bcf1-8d722cd56a6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7v5bn\" (UID: \"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:59.084408 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.084378 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6d84b\"" Jul 17 12:26:59.093251 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.093181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" Jul 17 12:26:59.223108 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.222921 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn"] Jul 17 12:26:59.225400 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:26:59.225369 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c95494b_39b9_4e8e_bcf1_8d722cd56a6f.slice/crio-a5fb5b34ebef68cc285f678f028d3a06dc5edbf37217319da162d77c90c643a1 WatchSource:0}: Error finding container a5fb5b34ebef68cc285f678f028d3a06dc5edbf37217319da162d77c90c643a1: Status 404 returned error can't find the container with id a5fb5b34ebef68cc285f678f028d3a06dc5edbf37217319da162d77c90c643a1 Jul 17 12:26:59.438536 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.438496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" event={"ID":"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f","Type":"ContainerStarted","Data":"a5fb5b34ebef68cc285f678f028d3a06dc5edbf37217319da162d77c90c643a1"} Jul 17 12:26:59.441635 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.441600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27zj" event={"ID":"ab7793de-4484-4e91-8ea9-6b32478a70fd","Type":"ContainerStarted","Data":"a641ecf120b87e855c55342f730e0fb0aca8351edfdb45664e7eb5ff99ddd31d"} Jul 17 12:26:59.459200 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:26:59.459146 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h27zj" podStartSLOduration=17.907141965 podStartE2EDuration="29.45909091s" totalImagesPullingTime="11.551948945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:26:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:59.457817898 +0000 UTC m=+269.402677405" watchObservedRunningTime="2026-07-17 12:26:59.45909091 +0000 UTC m=+269.403950413" Jul 17 12:27:01.449969 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:01.449935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" event={"ID":"2c95494b-39b9-4e8e-bcf1-8d722cd56a6f","Type":"ContainerStarted","Data":"d4b34228f2988e0cbd2dcc11423c5759808261d850e459809e3a583b000c3b4e"} Jul 17 12:27:01.474746 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:01.474670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7v5bn" podStartSLOduration=33.728473281 podStartE2EDuration="35.474654147s" totalImagesPullingTime="1.746180866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:01.473765286 +0000 UTC m=+271.418624798" watchObservedRunningTime="2026-07-17 12:27:01.474654147 +0000 UTC m=+271.419513650" Jul 17 12:27:02.292418 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:02.292384 2568 patch_prober.go:28] interesting pod/image-registry-6756c7dc49-pw49x container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 12:27:02.292582 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:02.292446 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" podUID="12a52a13-07ec-4c12-bf1d-698dc50ae6a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 12:27:04.385558 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:04.385522 2568 patch_prober.go:28] interesting pod/image-registry-6756c7dc49-pw49x container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 12:27:04.385925 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:04.385574 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" podUID="12a52a13-07ec-4c12-bf1d-698dc50ae6a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 12:27:07.636912 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.636874 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:27:07.639893 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.639877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.642250 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.642228 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 12:27:07.642353 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.642326 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 12:27:07.642407 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.642380 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 12:27:07.643286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.643263 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-28frs\"" Jul 17 12:27:07.643399 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.643353 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 12:27:07.643399 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.643387 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 12:27:07.643515 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.643402 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 12:27:07.643574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.643518 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 12:27:07.647380 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.647362 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 12:27:07.649952 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.649931 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:27:07.711552 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711507 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711755 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711755 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711755 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711755 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtdg\" (UniqueName: \"kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.711937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.711808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812871 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812985 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812985 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812985 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtdg\" (UniqueName: \"kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812985 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.812985 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.812979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.813224 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.813000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.813642 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.813618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.813830 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.813803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.813936 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.813887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.813936 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.813892 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.815544 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.815517 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.815640 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.815558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.820952 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.820922 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtdg\" (UniqueName: \"kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg\") pod \"console-7dcff46fc-gb89b\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:07.948904 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:07.948873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:08.079241 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:08.079208 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:27:08.082261 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:27:08.082228 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3a6b27_5e9f_4fc9_bb10_50106b0eb4c3.slice/crio-e2c0539850c55daf51747f49f435ae39793502f89cc862c8fc6647d861292ee0 WatchSource:0}: Error finding container e2c0539850c55daf51747f49f435ae39793502f89cc862c8fc6647d861292ee0: Status 404 returned error can't find the container with id e2c0539850c55daf51747f49f435ae39793502f89cc862c8fc6647d861292ee0 Jul 17 12:27:08.472640 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:08.472596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcff46fc-gb89b" event={"ID":"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3","Type":"ContainerStarted","Data":"e2c0539850c55daf51747f49f435ae39793502f89cc862c8fc6647d861292ee0"} Jul 17 12:27:11.483900 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:11.483860 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcff46fc-gb89b" event={"ID":"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3","Type":"ContainerStarted","Data":"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e"} Jul 17 12:27:11.502959 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:11.502911 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dcff46fc-gb89b" podStartSLOduration=1.933311546 podStartE2EDuration="4.502892968s" totalImagesPullingTime="2.569581422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:11.50192195 +0000 UTC m=+281.446781453" watchObservedRunningTime="2026-07-17 12:27:11.502892968 +0000 UTC m=+281.447752553" Jul 17 12:27:12.137960 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:12.137905 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n85zn" podUID="4f8dba38-59ce-4c23-8e17-5a713a852153" Jul 17 12:27:12.292161 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:12.292129 2568 patch_prober.go:28] interesting pod/image-registry-6756c7dc49-pw49x container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 12:27:12.292350 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:12.292200 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" podUID="12a52a13-07ec-4c12-bf1d-698dc50ae6a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 12:27:12.487321 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:12.487288 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:14.385909 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:14.385884 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6756c7dc49-pw49x" Jul 17 12:27:15.374653 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.374598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:15.377153 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.377127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8dba38-59ce-4c23-8e17-5a713a852153-metrics-tls\") pod \"dns-default-n85zn\" (UID: \"4f8dba38-59ce-4c23-8e17-5a713a852153\") " pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:15.490453 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.490418 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\"" Jul 17 12:27:15.499288 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.499259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:15.637921 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.637717 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n85zn"] Jul 17 12:27:15.640216 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:27:15.640190 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8dba38_59ce_4c23_8e17_5a713a852153.slice/crio-732740de1d64548cc6b5b4ea7f872a49ad70a2f3eb43ff6574645194b6839f14 WatchSource:0}: Error finding container 732740de1d64548cc6b5b4ea7f872a49ad70a2f3eb43ff6574645194b6839f14: Status 404 returned error can't find the container with id 732740de1d64548cc6b5b4ea7f872a49ad70a2f3eb43ff6574645194b6839f14 Jul 17 12:27:15.883839 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.883803 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-shxjx"] Jul 17 12:27:15.888128 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.888076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.891009 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.890985 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 12:27:15.891009 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.891000 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 12:27:15.891167 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.891080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rsxtw\"" Jul 17 12:27:15.891251 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.891235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 12:27:15.891479 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.891465 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 12:27:15.981087 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-textfile\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xth\" (UniqueName: \"kubernetes.io/projected/520ca06e-980e-4431-9736-355c74e80b74-kube-api-access-77xth\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981152 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-metrics-client-ca\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-root\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981289 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-accelerators-collector-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-sys\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:15.981486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:15.981387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-wtmp\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082004 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.081971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-wtmp\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082004 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082244 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-textfile\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082244 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xth\" (UniqueName: \"kubernetes.io/projected/520ca06e-980e-4431-9736-355c74e80b74-kube-api-access-77xth\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082244 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082170 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-wtmp\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082244 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-metrics-client-ca\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-root\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-accelerators-collector-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-sys\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082404 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-root\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082455 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:16.082439 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 12:27:16.082822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/520ca06e-980e-4431-9736-355c74e80b74-sys\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082822 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:16.082509 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls podName:520ca06e-980e-4431-9736-355c74e80b74 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:16.582488087 +0000 UTC m=+286.527347576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls") pod "node-exporter-shxjx" (UID: "520ca06e-980e-4431-9736-355c74e80b74") : secret "node-exporter-tls" not found Jul 17 12:27:16.082822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-textfile\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082923 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082884 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-metrics-client-ca\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.082973 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.082953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-accelerators-collector-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.084459 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.084433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.089807 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.089781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xth\" (UniqueName: \"kubernetes.io/projected/520ca06e-980e-4431-9736-355c74e80b74-kube-api-access-77xth\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.500140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.500097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n85zn" event={"ID":"4f8dba38-59ce-4c23-8e17-5a713a852153","Type":"ContainerStarted","Data":"732740de1d64548cc6b5b4ea7f872a49ad70a2f3eb43ff6574645194b6839f14"} Jul 17 12:27:16.586207 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.586173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.589136 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.589106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/520ca06e-980e-4431-9736-355c74e80b74-node-exporter-tls\") pod \"node-exporter-shxjx\" (UID: \"520ca06e-980e-4431-9736-355c74e80b74\") " pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.797787 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:16.797695 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-shxjx" Jul 17 12:27:16.808918 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:27:16.808892 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520ca06e_980e_4431_9736_355c74e80b74.slice/crio-1c4cd670c50d2a892f8b2429fc8b9d142eff7323ae94e917d59a61febfd91e18 WatchSource:0}: Error finding container 1c4cd670c50d2a892f8b2429fc8b9d142eff7323ae94e917d59a61febfd91e18: Status 404 returned error can't find the container with id 1c4cd670c50d2a892f8b2429fc8b9d142eff7323ae94e917d59a61febfd91e18 Jul 17 12:27:17.505783 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:17.505749 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shxjx" event={"ID":"520ca06e-980e-4431-9736-355c74e80b74","Type":"ContainerStarted","Data":"1c4cd670c50d2a892f8b2429fc8b9d142eff7323ae94e917d59a61febfd91e18"} Jul 17 12:27:17.950013 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:17.949971 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:17.950195 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:17.950043 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:17.955954 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:17.955924 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:18.511138 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.511097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n85zn" event={"ID":"4f8dba38-59ce-4c23-8e17-5a713a852153","Type":"ContainerStarted","Data":"8f9d93bc6ad840f81e4d54e07acdfbb7fd6d328e7954ef8f99ad9777e7d001c0"} Jul 17 12:27:18.511596 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.511157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n85zn" event={"ID":"4f8dba38-59ce-4c23-8e17-5a713a852153","Type":"ContainerStarted","Data":"80cabd83340b5ad078157efd2d581301aa5cf47a02deb3b84cc9718807aba271"} Jul 17 12:27:18.511596 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.511188 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:18.512587 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.512567 2568 generic.go:356] "Generic (PLEG): container finished" podID="520ca06e-980e-4431-9736-355c74e80b74" containerID="fd0941c8f6922952199ae12aac68ca7482d7e0ac8b648fb7c38a3998f179effb" exitCode=0 Jul 17 12:27:18.512668 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.512629 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shxjx" event={"ID":"520ca06e-980e-4431-9736-355c74e80b74","Type":"ContainerDied","Data":"fd0941c8f6922952199ae12aac68ca7482d7e0ac8b648fb7c38a3998f179effb"} Jul 17 12:27:18.516674 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.516652 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:27:18.528426 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:18.528369 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n85zn" podStartSLOduration=251.738209852 podStartE2EDuration="4m13.528350172s" totalImagesPullingTime="1.79014032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:18.52625864 +0000 UTC m=+288.471118143" watchObservedRunningTime="2026-07-17 12:27:18.528350172 +0000 UTC m=+288.473209680" Jul 17 12:27:19.517774 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:19.517736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shxjx" event={"ID":"520ca06e-980e-4431-9736-355c74e80b74","Type":"ContainerStarted","Data":"baf1a0365c9aa01262003ec5ee83987d93cac37328ebf5ddda371b892d4a4d2a"} Jul 17 12:27:19.518141 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:19.517793 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shxjx" event={"ID":"520ca06e-980e-4431-9736-355c74e80b74","Type":"ContainerStarted","Data":"ef2308d9dafa48421b0a1dbd7c47be6514694137a915ba38242571fe1c51487c"} Jul 17 12:27:19.536482 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:19.536433 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-shxjx" podStartSLOduration=3.246399382 podStartE2EDuration="4.536419401s" totalImagesPullingTime="1.219964187s" totalInitContainerRuntime="70.055832ms" isStatefulPod=true podCreationTimestamp="2026-07-17 12:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:19.535585162 +0000 UTC m=+289.480444666" watchObservedRunningTime="2026-07-17 12:27:19.536419401 +0000 UTC m=+289.481278905" Jul 17 12:27:20.656090 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.656054 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd"] Jul 17 12:27:20.682304 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.682274 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd"] Jul 17 12:27:20.682451 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.682363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:20.684394 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.684351 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 12:27:20.684523 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.684502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k8r2g\"" Jul 17 12:27:20.823992 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.823954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69tf\" (UniqueName: \"kubernetes.io/projected/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-kube-api-access-v69tf\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:20.823992 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.823995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:20.925020 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.924936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69tf\" (UniqueName: \"kubernetes.io/projected/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-kube-api-access-v69tf\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:20.925020 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.925000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:20.925196 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:20.925167 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 12:27:20.925255 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:20.925244 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert podName:eed48311-7549-43f3-ab61-4cd4a6d3e8a0 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:21.425225457 +0000 UTC m=+291.370084939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-nhchd" (UID: "eed48311-7549-43f3-ab61-4cd4a6d3e8a0") : secret "monitoring-plugin-cert" not found Jul 17 12:27:20.932992 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:20.932974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69tf\" (UniqueName: \"kubernetes.io/projected/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-kube-api-access-v69tf\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:21.428861 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:21.428828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:21.431272 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:21.431245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eed48311-7549-43f3-ab61-4cd4a6d3e8a0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-nhchd\" (UID: \"eed48311-7549-43f3-ab61-4cd4a6d3e8a0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:21.591166 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:21.591130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:21.769738 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:21.769675 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd"] Jul 17 12:27:21.772572 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:27:21.772547 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed48311_7549_43f3_ab61_4cd4a6d3e8a0.slice/crio-1738160b9f958ef52933fa17c7ce362ab795f822f17b29f5d0d64eede0198122 WatchSource:0}: Error finding container 1738160b9f958ef52933fa17c7ce362ab795f822f17b29f5d0d64eede0198122: Status 404 returned error can't find the container with id 1738160b9f958ef52933fa17c7ce362ab795f822f17b29f5d0d64eede0198122 Jul 17 12:27:22.107184 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.107101 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 12:27:22.142489 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.142459 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 12:27:22.142648 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.142573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.145243 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145223 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 12:27:22.145446 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145406 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 12:27:22.145446 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145407 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 12:27:22.145677 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145662 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 12:27:22.145783 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145757 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 12:27:22.145783 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145758 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 12:27:22.145960 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 12:27:22.145960 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.145891 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 12:27:22.146131 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146099 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 12:27:22.146131 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146119 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 12:27:22.146224 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146136 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-13ibr1s5qa60m\"" Jul 17 12:27:22.146224 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146184 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qcbjf\"" Jul 17 12:27:22.146224 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 12:27:22.146468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146434 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 12:27:22.146468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146437 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 12:27:22.146605 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.146477 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 12:27:22.151998 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.151973 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 12:27:22.336648 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336648 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336658 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-config-out\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.336914 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.336991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337103 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-web-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337173 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wck\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-kube-api-access-22wck\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337545 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337545 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337545 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.337545 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.337329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438282 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438282 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438306 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438403 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-web-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wck\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-kube-api-access-22wck\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438530 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-config-out\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.438974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.438778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.439520 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.439045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.439520 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.439475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.439779 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.439750 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.440294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.440271 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.440525 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.440500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.442437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.442526 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.443001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.443063 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bca664a3-30b5-4cce-9083-6cf5b9583620-config-out\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.443227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bca664a3-30b5-4cce-9083-6cf5b9583620-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.443507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.443894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.443631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.444305 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.444171 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.444305 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.444178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.446034 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.446010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.446443 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.446401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.446556 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.446534 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-web-config\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.446808 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.446776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.446808 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.446786 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wck\" (UniqueName: \"kubernetes.io/projected/bca664a3-30b5-4cce-9083-6cf5b9583620-kube-api-access-22wck\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.447228 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.447211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bca664a3-30b5-4cce-9083-6cf5b9583620-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bca664a3-30b5-4cce-9083-6cf5b9583620\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.457303 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.457280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:22.536769 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.536732 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" event={"ID":"eed48311-7549-43f3-ab61-4cd4a6d3e8a0","Type":"ContainerStarted","Data":"1738160b9f958ef52933fa17c7ce362ab795f822f17b29f5d0d64eede0198122"} Jul 17 12:27:22.631577 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:22.631545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 12:27:22.635309 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:27:22.635286 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca664a3_30b5_4cce_9083_6cf5b9583620.slice/crio-e638dbf2fe3a9a9f4779c6035e7742ea0406707e232bd9c751f19e37ac53e7eb WatchSource:0}: Error finding container e638dbf2fe3a9a9f4779c6035e7742ea0406707e232bd9c751f19e37ac53e7eb: Status 404 returned error can't find the container with id e638dbf2fe3a9a9f4779c6035e7742ea0406707e232bd9c751f19e37ac53e7eb Jul 17 12:27:23.541892 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:23.541851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"e638dbf2fe3a9a9f4779c6035e7742ea0406707e232bd9c751f19e37ac53e7eb"} Jul 17 12:27:23.543355 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:23.543326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" event={"ID":"eed48311-7549-43f3-ab61-4cd4a6d3e8a0","Type":"ContainerStarted","Data":"2591ee070a5d6c7c2668221e57feb305952ff2b34d4a915ddb41aba2caf0f174"} Jul 17 12:27:23.543598 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:23.543580 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:23.549848 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:23.549828 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" Jul 17 12:27:23.562654 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:23.562607 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-nhchd" podStartSLOduration=2.345238735 podStartE2EDuration="3.562586832s" totalImagesPullingTime="1.217348097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:23.561426356 +0000 UTC m=+293.506285863" watchObservedRunningTime="2026-07-17 12:27:23.562586832 +0000 UTC m=+293.507446339" Jul 17 12:27:24.549944 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:24.549914 2568 generic.go:356] "Generic (PLEG): container finished" podID="bca664a3-30b5-4cce-9083-6cf5b9583620" containerID="c7991153211b26f69dcf822acd2bcee112aa576b446aa74039ec1c20a65e3a91" exitCode=0 Jul 17 12:27:24.550461 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:24.550001 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerDied","Data":"c7991153211b26f69dcf822acd2bcee112aa576b446aa74039ec1c20a65e3a91"} Jul 17 12:27:26.582280 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:27:26.582245 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6t2j" podUID="3ae54f7a-49e7-4701-a8e8-af4f183d4d09" Jul 17 12:27:27.566080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:27.565837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"23b09ccff1dd21c724ec82591af6a82d1a45931e05cd289effce7310eb6d6a00"} Jul 17 12:27:27.566080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:27.565947 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"0c10475a3ee8618480005a66f962ffff123f1171dc737c96d4bb5db18add27d9"} Jul 17 12:27:28.520285 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:28.520257 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n85zn" Jul 17 12:27:30.462116 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.462041 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 12:27:30.584796 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.584634 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"fd857c4e1944bf7f9fca0678eeef2cd480dda74b888bcdb6f2acd20ff53ce24f"} Jul 17 12:27:30.584796 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.584716 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"faa234f12904709ffe682e4e65af3660dd4dbf623a80ab8064c57bbd09ab1fbf"} Jul 17 12:27:30.584796 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.584746 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"92494d14196eee39d327102b019d0661325bb36a3ee9be56ba52130ac4a979e0"} Jul 17 12:27:30.584796 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.584770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bca664a3-30b5-4cce-9083-6cf5b9583620","Type":"ContainerStarted","Data":"1c47c9e7c9df43e2ec241cade04e2c5b939153445495e905c071bd40e880cbe8"} Jul 17 12:27:30.613211 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:30.613123 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.452708932 podStartE2EDuration="8.613102961s" totalImagesPullingTime="6.141830312s" totalInitContainerRuntime="18.563717ms" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:30.612739884 +0000 UTC m=+300.557599388" watchObservedRunningTime="2026-07-17 12:27:30.613102961 +0000 UTC m=+300.557962465" Jul 17 12:27:32.458217 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:32.458166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:27:40.334869 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:40.334838 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:27:41.581474 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:41.581436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:27:55.677880 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:55.677843 2568 generic.go:356] "Generic (PLEG): container finished" podID="6468c778-c914-484e-83a9-eae7fe6f52d0" containerID="62a4237389c845c93910c8580bc015db8d9b49b7724c829009616bb6028d53ec" exitCode=0 Jul 17 12:27:55.678277 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:55.677909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" event={"ID":"6468c778-c914-484e-83a9-eae7fe6f52d0","Type":"ContainerDied","Data":"62a4237389c845c93910c8580bc015db8d9b49b7724c829009616bb6028d53ec"} Jul 17 12:27:55.678277 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:55.678204 2568 scope.go:117] "RemoveContainer" containerID="62a4237389c845c93910c8580bc015db8d9b49b7724c829009616bb6028d53ec" Jul 17 12:27:56.683074 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:27:56.683042 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cjcz9" event={"ID":"6468c778-c914-484e-83a9-eae7fe6f52d0","Type":"ContainerStarted","Data":"2795b9b4a3b6fc83fd3946d34ac4f4e1aa6b4af6c7187457622f9fd720d13a6b"} Jul 17 12:28:00.698767 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:00.698730 2568 generic.go:356] "Generic (PLEG): container finished" podID="37431f8a-bcda-4b4d-b8a2-6dc14a402a5f" containerID="d8ed1a3e4df52a30bfa67e8f2c47c09f745db14da0c2b0ccf896d2a31dfc3646" exitCode=0 Jul 17 12:28:00.699211 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:00.698801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" event={"ID":"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f","Type":"ContainerDied","Data":"d8ed1a3e4df52a30bfa67e8f2c47c09f745db14da0c2b0ccf896d2a31dfc3646"} Jul 17 12:28:00.699211 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:00.699099 2568 scope.go:117] "RemoveContainer" containerID="d8ed1a3e4df52a30bfa67e8f2c47c09f745db14da0c2b0ccf896d2a31dfc3646" Jul 17 12:28:01.703978 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:01.703946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-kl8f8" event={"ID":"37431f8a-bcda-4b4d-b8a2-6dc14a402a5f","Type":"ContainerStarted","Data":"f50ee9f29de94108797c7d7ae2d52b802adcd3d43aad3a38a23fe0f63bb7cd95"} Jul 17 12:28:05.359495 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.359433 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dcff46fc-gb89b" podUID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" containerName="console" containerID="cri-o://cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e" gracePeriod=15 Jul 17 12:28:05.606313 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.606290 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:28:05.626567 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626432 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgtdg\" (UniqueName: \"kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626567 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626477 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626567 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626522 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626856 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626585 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626856 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626622 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626856 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626660 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.626856 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626707 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert\") pod \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\" (UID: \"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3\") " Jul 17 12:28:05.627047 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.626913 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 12:28:05.627047 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.627036 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-service-ca\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.627286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.627261 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 12:28:05.627286 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.627273 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 12:28:05.627665 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.627634 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config" (OuterVolumeSpecName: "console-config") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 12:28:05.629508 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.629470 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg" (OuterVolumeSpecName: "kube-api-access-xgtdg") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "kube-api-access-xgtdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 12:28:05.629613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.629510 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 12:28:05.629613 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.629551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" (UID: "0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 12:28:05.719977 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.719937 2568 generic.go:356] "Generic (PLEG): container finished" podID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" containerID="cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e" exitCode=0 Jul 17 12:28:05.720124 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.719990 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcff46fc-gb89b" Jul 17 12:28:05.720124 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.720015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcff46fc-gb89b" event={"ID":"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3","Type":"ContainerDied","Data":"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e"} Jul 17 12:28:05.720124 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.720064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcff46fc-gb89b" event={"ID":"0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3","Type":"ContainerDied","Data":"e2c0539850c55daf51747f49f435ae39793502f89cc862c8fc6647d861292ee0"} Jul 17 12:28:05.720906 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.720885 2568 scope.go:117] "RemoveContainer" containerID="cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727639 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-oauth-serving-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727658 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgtdg\" (UniqueName: \"kubernetes.io/projected/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-kube-api-access-xgtdg\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727669 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-config\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727678 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-trusted-ca-bundle\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727707 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-serving-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.727792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.727715 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3-console-oauth-config\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:28:05.729511 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.729493 2568 scope.go:117] "RemoveContainer" containerID="cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e" Jul 17 12:28:05.729776 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:28:05.729757 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e\": container with ID starting with cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e not found: ID does not exist" containerID="cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e" Jul 17 12:28:05.729816 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.729785 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e"} err="failed to get container status \"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e\": rpc error: code = NotFound desc = could not find container \"cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e\": container with ID starting with cff1516e887846891d01cb05ff7d45fdd27dbc87840d1dc9ae9f39b7e4bf2d1e not found: ID does not exist" Jul 17 12:28:05.740105 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.740081 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:28:05.744747 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:05.744727 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dcff46fc-gb89b"] Jul 17 12:28:06.585424 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:06.585394 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" path="/var/lib/kubelet/pods/0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3/volumes" Jul 17 12:28:22.458391 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:22.458307 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:28:22.475042 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:22.475015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:28:22.800877 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:22.800807 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 12:28:44.360372 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.360322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:28:44.362726 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.362679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae54f7a-49e7-4701-a8e8-af4f183d4d09-metrics-certs\") pod \"network-metrics-daemon-l6t2j\" (UID: \"3ae54f7a-49e7-4701-a8e8-af4f183d4d09\") " pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:28:44.584073 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.584042 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\"" Jul 17 12:28:44.592966 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.592934 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6t2j" Jul 17 12:28:44.721538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.721503 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6t2j"] Jul 17 12:28:44.724482 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:28:44.724453 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae54f7a_49e7_4701_a8e8_af4f183d4d09.slice/crio-ec605c1bd27bf0ab6c5deec018b735b0ea14892065df27449bb36b2754a4441d WatchSource:0}: Error finding container ec605c1bd27bf0ab6c5deec018b735b0ea14892065df27449bb36b2754a4441d: Status 404 returned error can't find the container with id ec605c1bd27bf0ab6c5deec018b735b0ea14892065df27449bb36b2754a4441d Jul 17 12:28:44.726523 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.726505 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 12:28:44.863623 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:44.863582 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6t2j" event={"ID":"3ae54f7a-49e7-4701-a8e8-af4f183d4d09","Type":"ContainerStarted","Data":"ec605c1bd27bf0ab6c5deec018b735b0ea14892065df27449bb36b2754a4441d"} Jul 17 12:28:45.868904 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:45.868865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6t2j" event={"ID":"3ae54f7a-49e7-4701-a8e8-af4f183d4d09","Type":"ContainerStarted","Data":"1e492ba3f951a9a97f3ee11a89bf94c487b8c689802b6dd04f4e116e16fe3c73"} Jul 17 12:28:45.869281 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:45.868963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6t2j" event={"ID":"3ae54f7a-49e7-4701-a8e8-af4f183d4d09","Type":"ContainerStarted","Data":"fec0f82090cbe54a8f2e9349f0c801567488cd6de9d137361e06c667c0ca7a1a"} Jul 17 12:28:45.885824 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:28:45.885760 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l6t2j" podStartSLOduration=373.947621298 podStartE2EDuration="6m14.885742285s" totalImagesPullingTime="938.120987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:28:45.884158273 +0000 UTC m=+375.829017775" watchObservedRunningTime="2026-07-17 12:28:45.885742285 +0000 UTC m=+375.830601781" Jul 17 12:41:24.221820 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.221780 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7"] Jul 17 12:41:24.222279 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.222068 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" containerName="console" Jul 17 12:41:24.222279 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.222079 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" containerName="console" Jul 17 12:41:24.222279 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.222138 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3a6b27-5e9f-4fc9-bb10-50106b0eb4c3" containerName="console" Jul 17 12:41:24.225086 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.225069 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.228014 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.227990 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 12:41:24.228014 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.228009 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p95ns\"" Jul 17 12:41:24.228202 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.227996 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 12:41:24.228202 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.228048 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 12:41:24.228202 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.228042 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 12:41:24.240363 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.240338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7"] Jul 17 12:41:24.306675 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.306635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.306894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.306734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.306894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.306753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.306894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.306776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.306894 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.306801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvm9\" (UniqueName: \"kubernetes.io/projected/62e054db-eec7-49e2-8646-b6666ac3bafa-kube-api-access-lsvm9\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408091 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408091 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408345 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408345 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvm9\" (UniqueName: \"kubernetes.io/projected/62e054db-eec7-49e2-8646-b6666ac3bafa-kube-api-access-lsvm9\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408345 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408255 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.408545 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.408521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.410525 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.410502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62e054db-eec7-49e2-8646-b6666ac3bafa-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.410738 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.410722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.410794 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.410735 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62e054db-eec7-49e2-8646-b6666ac3bafa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.416016 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.415990 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvm9\" (UniqueName: \"kubernetes.io/projected/62e054db-eec7-49e2-8646-b6666ac3bafa-kube-api-access-lsvm9\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-sn6f7\" (UID: \"62e054db-eec7-49e2-8646-b6666ac3bafa\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.535946 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.535860 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:24.669439 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.669353 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7"] Jul 17 12:41:24.672226 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:41:24.672199 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e054db_eec7_49e2_8646_b6666ac3bafa.slice/crio-f90b87207c2cbab0335b5e7003a7f953fca66d5930be6fafffce4df8b6c812de WatchSource:0}: Error finding container f90b87207c2cbab0335b5e7003a7f953fca66d5930be6fafffce4df8b6c812de: Status 404 returned error can't find the container with id f90b87207c2cbab0335b5e7003a7f953fca66d5930be6fafffce4df8b6c812de Jul 17 12:41:24.674093 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:24.674077 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 12:41:25.551319 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:25.551229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" event={"ID":"62e054db-eec7-49e2-8646-b6666ac3bafa","Type":"ContainerStarted","Data":"f90b87207c2cbab0335b5e7003a7f953fca66d5930be6fafffce4df8b6c812de"} Jul 17 12:41:27.559161 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:27.559127 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" event={"ID":"62e054db-eec7-49e2-8646-b6666ac3bafa","Type":"ContainerStarted","Data":"366aab7f6aea76c174dadd73ac21ee365953611f4d7f87d28938b4344856ae0e"} Jul 17 12:41:29.567412 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:29.567375 2568 generic.go:356] "Generic (PLEG): container finished" podID="62e054db-eec7-49e2-8646-b6666ac3bafa" containerID="366aab7f6aea76c174dadd73ac21ee365953611f4d7f87d28938b4344856ae0e" exitCode=0 Jul 17 12:41:29.567847 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:29.567438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" event={"ID":"62e054db-eec7-49e2-8646-b6666ac3bafa","Type":"ContainerDied","Data":"366aab7f6aea76c174dadd73ac21ee365953611f4d7f87d28938b4344856ae0e"} Jul 17 12:41:30.572287 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:30.572251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" event={"ID":"62e054db-eec7-49e2-8646-b6666ac3bafa","Type":"ContainerStarted","Data":"533acd357561be0f21c936af787903cad909154bd1df450983e8b58b291fe6c1"} Jul 17 12:41:30.572672 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:30.572470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:30.595324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:30.595272 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" podStartSLOduration=2.404998925 podStartE2EDuration="6.595257389s" totalImagesPullingTime="3.153457454s" totalInitContainerRuntime="1.03680101s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:30.593217896 +0000 UTC m=+1140.538077399" watchObservedRunningTime="2026-07-17 12:41:30.595257389 +0000 UTC m=+1140.540116891" Jul 17 12:41:41.577956 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:41.577928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-sn6f7" Jul 17 12:41:43.742639 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.742601 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg"] Jul 17 12:41:43.754128 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.754100 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.757833 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.757812 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 12:41:43.758497 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.758465 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cx57d\"" Jul 17 12:41:43.758599 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.758498 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 12:41:43.759858 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.759835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg"] Jul 17 12:41:43.775927 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.771150 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f6db39-82a4-4bff-9875-ab666896b5c2-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.775927 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.771199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6twf\" (UniqueName: \"kubernetes.io/projected/a8f6db39-82a4-4bff-9875-ab666896b5c2-kube-api-access-z6twf\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.775927 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.771251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f6db39-82a4-4bff-9875-ab666896b5c2-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.872150 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.872093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f6db39-82a4-4bff-9875-ab666896b5c2-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.872150 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.872152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6twf\" (UniqueName: \"kubernetes.io/projected/a8f6db39-82a4-4bff-9875-ab666896b5c2-kube-api-access-z6twf\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.872406 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.872191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f6db39-82a4-4bff-9875-ab666896b5c2-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.874498 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.874468 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f6db39-82a4-4bff-9875-ab666896b5c2-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.874607 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.874597 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f6db39-82a4-4bff-9875-ab666896b5c2-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.886218 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.886193 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6twf\" (UniqueName: \"kubernetes.io/projected/a8f6db39-82a4-4bff-9875-ab666896b5c2-kube-api-access-z6twf\") pod \"kube-auth-proxy-6bfcdd9cc7-mg7pg\" (UID: \"a8f6db39-82a4-4bff-9875-ab666896b5c2\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:43.969516 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.969483 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lsnqt"] Jul 17 12:41:43.971976 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.971961 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:43.974326 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.974303 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2sb9z\"" Jul 17 12:41:43.974538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.974519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 17 12:41:43.984044 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:43.984022 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lsnqt"] Jul 17 12:41:44.065334 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.065255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" Jul 17 12:41:44.073132 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.073110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j45c\" (UniqueName: \"kubernetes.io/projected/04716e68-f3ee-42ca-9778-c2f34c791ac4-kube-api-access-9j45c\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.073254 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.073153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.174370 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.174339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j45c\" (UniqueName: \"kubernetes.io/projected/04716e68-f3ee-42ca-9778-c2f34c791ac4-kube-api-access-9j45c\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.174370 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.174378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.174549 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:41:44.174496 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 17 12:41:44.174581 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:41:44.174550 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert podName:04716e68-f3ee-42ca-9778-c2f34c791ac4 nodeName:}" failed. No retries permitted until 2026-07-17 12:41:44.674531731 +0000 UTC m=+1154.619391230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert") pod "odh-model-controller-7998c67dc4-lsnqt" (UID: "04716e68-f3ee-42ca-9778-c2f34c791ac4") : secret "odh-model-controller-webhook-cert" not found Jul 17 12:41:44.187465 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.187438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j45c\" (UniqueName: \"kubernetes.io/projected/04716e68-f3ee-42ca-9778-c2f34c791ac4-kube-api-access-9j45c\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.196819 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.196797 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg"] Jul 17 12:41:44.199460 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:41:44.199436 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f6db39_82a4_4bff_9875_ab666896b5c2.slice/crio-c3d6bd46940d9fae7aa31e2a9a88d78f14bdb0128e22d2a73ce7630e2ba74d93 WatchSource:0}: Error finding container c3d6bd46940d9fae7aa31e2a9a88d78f14bdb0128e22d2a73ce7630e2ba74d93: Status 404 returned error can't find the container with id c3d6bd46940d9fae7aa31e2a9a88d78f14bdb0128e22d2a73ce7630e2ba74d93 Jul 17 12:41:44.622321 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.622290 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" event={"ID":"a8f6db39-82a4-4bff-9875-ab666896b5c2","Type":"ContainerStarted","Data":"c3d6bd46940d9fae7aa31e2a9a88d78f14bdb0128e22d2a73ce7630e2ba74d93"} Jul 17 12:41:44.678578 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.678548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.680942 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.680919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04716e68-f3ee-42ca-9778-c2f34c791ac4-cert\") pod \"odh-model-controller-7998c67dc4-lsnqt\" (UID: \"04716e68-f3ee-42ca-9778-c2f34c791ac4\") " pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:44.881550 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:44.881467 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:45.029079 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:45.029047 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lsnqt"] Jul 17 12:41:45.031639 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:41:45.031609 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04716e68_f3ee_42ca_9778_c2f34c791ac4.slice/crio-dc916c0d3332ad0af667edfa430cbc30261031387ddd199fa0fdbb1e8836e1db WatchSource:0}: Error finding container dc916c0d3332ad0af667edfa430cbc30261031387ddd199fa0fdbb1e8836e1db: Status 404 returned error can't find the container with id dc916c0d3332ad0af667edfa430cbc30261031387ddd199fa0fdbb1e8836e1db Jul 17 12:41:45.628531 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:45.628498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" event={"ID":"04716e68-f3ee-42ca-9778-c2f34c791ac4","Type":"ContainerStarted","Data":"dc916c0d3332ad0af667edfa430cbc30261031387ddd199fa0fdbb1e8836e1db"} Jul 17 12:41:47.638254 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:47.638222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" event={"ID":"a8f6db39-82a4-4bff-9875-ab666896b5c2","Type":"ContainerStarted","Data":"b450bc96a3a54aa9c4f5a78553ec74f9e6754b88a74d69589a4f15c7c45d33cf"} Jul 17 12:41:47.654661 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:47.654610 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-mg7pg" podStartSLOduration=1.945745678 podStartE2EDuration="4.654595206s" totalImagesPullingTime="2.708849528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:47.653507776 +0000 UTC m=+1157.598367280" watchObservedRunningTime="2026-07-17 12:41:47.654595206 +0000 UTC m=+1157.599454708" Jul 17 12:41:48.645174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:48.644888 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" event={"ID":"04716e68-f3ee-42ca-9778-c2f34c791ac4","Type":"ContainerStarted","Data":"b5f2120566a6e32c2ce880df9110d5ba410273e8ca0f17f851e7fd9d5327d7b9"} Jul 17 12:41:48.645174 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:48.645121 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:41:48.673696 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:48.673641 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" podStartSLOduration=2.495964511 podStartE2EDuration="5.673626475s" totalImagesPullingTime="3.177661964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:48.671531414 +0000 UTC m=+1158.616390916" watchObservedRunningTime="2026-07-17 12:41:48.673626475 +0000 UTC m=+1158.618485977" Jul 17 12:41:50.298830 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.298801 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h"] Jul 17 12:41:50.302074 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.302055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.305206 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.305186 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 12:41:50.305294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.305186 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xj2p5\"" Jul 17 12:41:50.317247 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.317219 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h"] Jul 17 12:41:50.328501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328475 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328618 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328512 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328618 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328618 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.328811 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.328759 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v54\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-kube-api-access-45v54\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429185 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429627 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429627 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429627 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v54\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-kube-api-access-45v54\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.429927 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429901 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.430193 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.429936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.430193 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.430125 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.431574 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.431553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.431792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.431776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.445836 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.445814 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.446026 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.446010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v54\" (UniqueName: \"kubernetes.io/projected/b0b59f2a-c9d4-4335-9e56-838ecd19a270-kube-api-access-45v54\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cxw4h\" (UID: \"b0b59f2a-c9d4-4335-9e56-838ecd19a270\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.613974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.613892 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:50.742890 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:50.742859 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h"] Jul 17 12:41:50.745778 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:41:50.745750 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b59f2a_c9d4_4335_9e56_838ecd19a270.slice/crio-9b1de1d33693c5f5ed0503372d001b107431e198512ec688a0ab7ed32cdfa0dd WatchSource:0}: Error finding container 9b1de1d33693c5f5ed0503372d001b107431e198512ec688a0ab7ed32cdfa0dd: Status 404 returned error can't find the container with id 9b1de1d33693c5f5ed0503372d001b107431e198512ec688a0ab7ed32cdfa0dd Jul 17 12:41:51.658546 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.658467 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" event={"ID":"b0b59f2a-c9d4-4335-9e56-838ecd19a270","Type":"ContainerStarted","Data":"9b1de1d33693c5f5ed0503372d001b107431e198512ec688a0ab7ed32cdfa0dd"} Jul 17 12:41:51.730658 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.730624 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69"] Jul 17 12:41:51.735031 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.735015 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.737260 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.737237 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 12:41:51.738127 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.738109 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8sx92\"" Jul 17 12:41:51.738184 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.738131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 12:41:51.738391 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.738373 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 12:41:51.738495 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.738405 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 12:41:51.738495 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.738408 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 12:41:51.742586 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.742544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69"] Jul 17 12:41:51.843252 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.843217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05946b95-63c3-4717-b8d2-19a92a65cf15-manager-config\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.843252 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.843251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.843486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.843288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdtz\" (UniqueName: \"kubernetes.io/projected/05946b95-63c3-4717-b8d2-19a92a65cf15-kube-api-access-sfdtz\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.843486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.843377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.944122 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.944035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.944122 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.944090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05946b95-63c3-4717-b8d2-19a92a65cf15-manager-config\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.944122 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.944113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.944336 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.944164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdtz\" (UniqueName: \"kubernetes.io/projected/05946b95-63c3-4717-b8d2-19a92a65cf15-kube-api-access-sfdtz\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.944880 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.944816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05946b95-63c3-4717-b8d2-19a92a65cf15-manager-config\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.946565 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.946544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.946651 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.946602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05946b95-63c3-4717-b8d2-19a92a65cf15-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:51.955144 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:51.955120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdtz\" (UniqueName: \"kubernetes.io/projected/05946b95-63c3-4717-b8d2-19a92a65cf15-kube-api-access-sfdtz\") pod \"lws-controller-manager-774b4c8dcd-b9j69\" (UID: \"05946b95-63c3-4717-b8d2-19a92a65cf15\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:52.045300 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:52.045259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:52.172071 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:52.172036 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69"] Jul 17 12:41:52.175055 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:41:52.175027 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05946b95_63c3_4717_b8d2_19a92a65cf15.slice/crio-4f8d32fc3234aad1dc053b213bfcb50fe587a78dd33be4c04edec078212ac7c2 WatchSource:0}: Error finding container 4f8d32fc3234aad1dc053b213bfcb50fe587a78dd33be4c04edec078212ac7c2: Status 404 returned error can't find the container with id 4f8d32fc3234aad1dc053b213bfcb50fe587a78dd33be4c04edec078212ac7c2 Jul 17 12:41:52.663225 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:52.663191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" event={"ID":"05946b95-63c3-4717-b8d2-19a92a65cf15","Type":"ContainerStarted","Data":"4f8d32fc3234aad1dc053b213bfcb50fe587a78dd33be4c04edec078212ac7c2"} Jul 17 12:41:54.675749 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:54.675707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" event={"ID":"05946b95-63c3-4717-b8d2-19a92a65cf15","Type":"ContainerStarted","Data":"6b006e4e476cddfe1b12dc2a3731de7348c5eaefaed7e66a0f1e9625c009df41"} Jul 17 12:41:54.676215 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:54.675855 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:41:54.696035 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:54.695978 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" podStartSLOduration=1.506463557 podStartE2EDuration="3.69596165s" totalImagesPullingTime="2.189498093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:54.694047759 +0000 UTC m=+1164.638907266" watchObservedRunningTime="2026-07-17 12:41:54.69596165 +0000 UTC m=+1164.640821153" Jul 17 12:41:56.289642 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:56.289608 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:41:56.289913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:56.289710 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:41:56.289913 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:56.289752 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:41:56.686084 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:56.686047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" event={"ID":"b0b59f2a-c9d4-4335-9e56-838ecd19a270","Type":"ContainerStarted","Data":"c0f8bfb5ec764deaa43645c08e9b827621cb81b87e2bac9152e86e84b79c03d4"} Jul 17 12:41:56.706935 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:56.706886 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" podStartSLOduration=1.165054921 podStartE2EDuration="6.706869023s" totalImagesPullingTime="5.541814102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:56.705004011 +0000 UTC m=+1166.649863513" watchObservedRunningTime="2026-07-17 12:41:56.706869023 +0000 UTC m=+1166.651728525" Jul 17 12:41:57.614968 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:57.614930 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:57.619631 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:57.619608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:57.690339 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:57.690310 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:57.691196 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:57.691179 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cxw4h" Jul 17 12:41:59.651091 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:41:59.651064 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lsnqt" Jul 17 12:42:05.683400 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:42:05.683370 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-b9j69" Jul 17 12:44:29.022125 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.022096 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk"] Jul 17 12:44:29.024776 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.024754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.027293 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.027271 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-k2lqx\"" Jul 17 12:44:29.036501 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.036479 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk"] Jul 17 12:44:29.052205 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4809646-3859-4172-ae02-d2ce581c3c10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4809646-3859-4172-ae02-d2ce581c3c10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052374 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkw7\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-kube-api-access-snkw7\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052645 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052645 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.052645 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.052467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153608 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4809646-3859-4172-ae02-d2ce581c3c10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkw7\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-kube-api-access-snkw7\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.153822 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154062 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4809646-3859-4172-ae02-d2ce581c3c10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154062 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154062 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.153887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154207 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.154161 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154207 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.154191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154332 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.154309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154421 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.154401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.154494 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.154475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4809646-3859-4172-ae02-d2ce581c3c10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.156139 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.156120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4809646-3859-4172-ae02-d2ce581c3c10-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.156330 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.156312 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4809646-3859-4172-ae02-d2ce581c3c10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.162540 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.162502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkw7\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-kube-api-access-snkw7\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.162982 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.162957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4809646-3859-4172-ae02-d2ce581c3c10-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-gqxlk\" (UID: \"b4809646-3859-4172-ae02-d2ce581c3c10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.334520 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.334442 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:29.465673 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.465642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk"] Jul 17 12:44:29.468160 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:44:29.468132 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4809646_3859_4172_ae02_d2ce581c3c10.slice/crio-3b25027456bf0e621f92c7c3853ea631cae76fbed3c2ab2e5464264081983796 WatchSource:0}: Error finding container 3b25027456bf0e621f92c7c3853ea631cae76fbed3c2ab2e5464264081983796: Status 404 returned error can't find the container with id 3b25027456bf0e621f92c7c3853ea631cae76fbed3c2ab2e5464264081983796 Jul 17 12:44:29.470213 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.470183 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:44:29.470298 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.470257 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:44:29.470359 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:29.470301 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:44:30.302381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:30.302345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" event={"ID":"b4809646-3859-4172-ae02-d2ce581c3c10","Type":"ContainerStarted","Data":"c885e864b93a63c8730844ab05189c9a5b8be95c2749f4a2216c7d8ff23dd1d9"} Jul 17 12:44:30.302786 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:30.302397 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" event={"ID":"b4809646-3859-4172-ae02-d2ce581c3c10","Type":"ContainerStarted","Data":"3b25027456bf0e621f92c7c3853ea631cae76fbed3c2ab2e5464264081983796"} Jul 17 12:44:30.321761 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:30.321699 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" podStartSLOduration=1.32166937 podStartE2EDuration="1.32166937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:30.320088832 +0000 UTC m=+1320.264948335" watchObservedRunningTime="2026-07-17 12:44:30.32166937 +0000 UTC m=+1320.266528873" Jul 17 12:44:30.335017 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:30.334990 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:30.339665 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:30.339645 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:31.306559 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:31.306530 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:31.307403 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:31.307387 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:44:32.149756 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.149726 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:32.152324 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.152306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:32.154769 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.154752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvgsb\"" Jul 17 12:44:32.154838 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.154766 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 12:44:32.155541 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.155526 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 12:44:32.158779 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.158537 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:32.180619 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.180585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh\") pod \"authorino-58cd955675-8nknb\" (UID: \"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3\") " pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:32.281879 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.281851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh\") pod \"authorino-58cd955675-8nknb\" (UID: \"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3\") " pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:32.289753 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.289731 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh\") pod \"authorino-58cd955675-8nknb\" (UID: \"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3\") " pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:32.470364 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.470293 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:32.594928 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:32.594905 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:32.596737 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:44:32.596708 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d1a82b_3e66_4fdf_bc7e_5a23b6e1b7d3.slice/crio-6d3fb133952c156cf1de4a670d169381cdc7af88e120b559989615ff9b152500 WatchSource:0}: Error finding container 6d3fb133952c156cf1de4a670d169381cdc7af88e120b559989615ff9b152500: Status 404 returned error can't find the container with id 6d3fb133952c156cf1de4a670d169381cdc7af88e120b559989615ff9b152500 Jul 17 12:44:33.317676 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:33.317624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8nknb" event={"ID":"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3","Type":"ContainerStarted","Data":"6d3fb133952c156cf1de4a670d169381cdc7af88e120b559989615ff9b152500"} Jul 17 12:44:36.334792 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:36.334752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8nknb" event={"ID":"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3","Type":"ContainerStarted","Data":"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb"} Jul 17 12:44:36.348695 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:36.348649 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8nknb" podStartSLOduration=1.236249322 podStartE2EDuration="4.348609454s" totalImagesPullingTime="3.112360132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:36.34816203 +0000 UTC m=+1326.293021534" watchObservedRunningTime="2026-07-17 12:44:36.348609454 +0000 UTC m=+1326.293468956" Jul 17 12:44:36.797620 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:36.797592 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:38.343299 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:38.343262 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8nknb" podUID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" containerName="authorino" containerID="cri-o://114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb" gracePeriod=30 Jul 17 12:44:38.584429 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:38.584406 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:38.642403 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:38.642327 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh\") pod \"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3\" (UID: \"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3\") " Jul 17 12:44:38.644538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:38.644515 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh" (OuterVolumeSpecName: "kube-api-access-w2kzh") pod "e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" (UID: "e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3"). InnerVolumeSpecName "kube-api-access-w2kzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 12:44:38.743184 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:38.743158 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3-kube-api-access-w2kzh\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:44:39.348721 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.348669 2568 generic.go:356] "Generic (PLEG): container finished" podID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" containerID="114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb" exitCode=0 Jul 17 12:44:39.348721 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.348729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8nknb" event={"ID":"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3","Type":"ContainerDied","Data":"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb"} Jul 17 12:44:39.349268 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.348746 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8nknb" Jul 17 12:44:39.349268 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.348758 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8nknb" event={"ID":"e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3","Type":"ContainerDied","Data":"6d3fb133952c156cf1de4a670d169381cdc7af88e120b559989615ff9b152500"} Jul 17 12:44:39.349268 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.348778 2568 scope.go:117] "RemoveContainer" containerID="114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb" Jul 17 12:44:39.357116 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.357096 2568 scope.go:117] "RemoveContainer" containerID="114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb" Jul 17 12:44:39.357347 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:44:39.357325 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb\": container with ID starting with 114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb not found: ID does not exist" containerID="114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb" Jul 17 12:44:39.357381 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.357355 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb"} err="failed to get container status \"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb\": rpc error: code = NotFound desc = could not find container \"114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb\": container with ID starting with 114d33b65073b74bc6d90bd76e2f1acd5d4e21e8123fc1d86c723f5de968d6bb not found: ID does not exist" Jul 17 12:44:39.368231 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.368210 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:39.372260 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:39.372242 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8nknb"] Jul 17 12:44:40.586061 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:44:40.586032 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" path="/var/lib/kubelet/pods/e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3/volumes" Jul 17 12:45:05.473378 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.473289 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:05.473937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.473754 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" containerName="authorino" Jul 17 12:45:05.473937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.473771 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" containerName="authorino" Jul 17 12:45:05.473937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.473876 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d1a82b-3e66-4fdf-bc7e-5a23b6e1b7d3" containerName="authorino" Jul 17 12:45:05.477448 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.477428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:05.479836 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.479816 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 12:45:05.480734 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.480719 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvgsb\"" Jul 17 12:45:05.480794 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.480726 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 12:45:05.488759 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.487618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:05.592092 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.592058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zncmz\" (UniqueName: \"kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz\") pod \"authorino-5d4b67c446-d7vqt\" (UID: \"740cda97-9bb7-46fe-a911-5dab641fa269\") " pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:05.692852 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.692813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zncmz\" (UniqueName: \"kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz\") pod \"authorino-5d4b67c446-d7vqt\" (UID: \"740cda97-9bb7-46fe-a911-5dab641fa269\") " pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:05.700669 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.700637 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zncmz\" (UniqueName: \"kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz\") pod \"authorino-5d4b67c446-d7vqt\" (UID: \"740cda97-9bb7-46fe-a911-5dab641fa269\") " pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:05.707728 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.707678 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:05.707937 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.707925 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:05.856616 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.856594 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:05.859134 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:45:05.859101 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod740cda97_9bb7_46fe_a911_5dab641fa269.slice/crio-7698177bb8309e81be92a3daf7a8e0c85d843655d9c035b8dde4204ff6e12d03 WatchSource:0}: Error finding container 7698177bb8309e81be92a3daf7a8e0c85d843655d9c035b8dde4204ff6e12d03: Status 404 returned error can't find the container with id 7698177bb8309e81be92a3daf7a8e0c85d843655d9c035b8dde4204ff6e12d03 Jul 17 12:45:05.938669 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.938638 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:45:05.956309 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.956282 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:45:05.956465 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.956395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:05.958796 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.958778 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 12:45:05.995429 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.995374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:05.995532 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:05.995447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5hn6\" (UniqueName: \"kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.096357 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.096323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5hn6\" (UniqueName: \"kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.096478 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.096397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.098871 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.098852 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.104618 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.104594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5hn6\" (UniqueName: \"kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6\") pod \"authorino-6f9884949-8k59k\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.267601 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.267525 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:45:06.402539 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.402501 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:45:06.405124 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:45:06.405099 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod486b5ace_7d2c_4c00_8f44_45b9326b15f9.slice/crio-d230096613b70c90d40a6627c6801650325a968ec35175c93723127a581f98dc WatchSource:0}: Error finding container d230096613b70c90d40a6627c6801650325a968ec35175c93723127a581f98dc: Status 404 returned error can't find the container with id d230096613b70c90d40a6627c6801650325a968ec35175c93723127a581f98dc Jul 17 12:45:06.456950 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.456914 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f9884949-8k59k" event={"ID":"486b5ace-7d2c-4c00-8f44-45b9326b15f9","Type":"ContainerStarted","Data":"d230096613b70c90d40a6627c6801650325a968ec35175c93723127a581f98dc"} Jul 17 12:45:06.457907 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:06.457885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" event={"ID":"740cda97-9bb7-46fe-a911-5dab641fa269","Type":"ContainerStarted","Data":"7698177bb8309e81be92a3daf7a8e0c85d843655d9c035b8dde4204ff6e12d03"} Jul 17 12:45:07.464593 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.464503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f9884949-8k59k" event={"ID":"486b5ace-7d2c-4c00-8f44-45b9326b15f9","Type":"ContainerStarted","Data":"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4"} Jul 17 12:45:07.466092 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.466066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" event={"ID":"740cda97-9bb7-46fe-a911-5dab641fa269","Type":"ContainerStarted","Data":"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8"} Jul 17 12:45:07.466219 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.466124 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" podUID="740cda97-9bb7-46fe-a911-5dab641fa269" containerName="authorino" containerID="cri-o://fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8" gracePeriod=30 Jul 17 12:45:07.481551 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.481509 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f9884949-8k59k" podStartSLOduration=1.715572478 podStartE2EDuration="2.481492445s" totalImagesPullingTime="765.919967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:07.47981776 +0000 UTC m=+1357.424677262" watchObservedRunningTime="2026-07-17 12:45:07.481492445 +0000 UTC m=+1357.426351978" Jul 17 12:45:07.495533 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.495481 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" podStartSLOduration=1.414887125 podStartE2EDuration="2.495460656s" totalImagesPullingTime="1.080573531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:07.494567585 +0000 UTC m=+1357.439427083" watchObservedRunningTime="2026-07-17 12:45:07.495460656 +0000 UTC m=+1357.440320159" Jul 17 12:45:07.693228 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.693205 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:07.809619 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.809544 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zncmz\" (UniqueName: \"kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz\") pod \"740cda97-9bb7-46fe-a911-5dab641fa269\" (UID: \"740cda97-9bb7-46fe-a911-5dab641fa269\") " Jul 17 12:45:07.811552 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.811532 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz" (OuterVolumeSpecName: "kube-api-access-zncmz") pod "740cda97-9bb7-46fe-a911-5dab641fa269" (UID: "740cda97-9bb7-46fe-a911-5dab641fa269"). InnerVolumeSpecName "kube-api-access-zncmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 12:45:07.910420 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:07.910389 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zncmz\" (UniqueName: \"kubernetes.io/projected/740cda97-9bb7-46fe-a911-5dab641fa269-kube-api-access-zncmz\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:45:08.472309 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.472274 2568 generic.go:356] "Generic (PLEG): container finished" podID="740cda97-9bb7-46fe-a911-5dab641fa269" containerID="fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8" exitCode=0 Jul 17 12:45:08.472309 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.472310 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" Jul 17 12:45:08.472753 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.472353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" event={"ID":"740cda97-9bb7-46fe-a911-5dab641fa269","Type":"ContainerDied","Data":"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8"} Jul 17 12:45:08.472753 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.472386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d7vqt" event={"ID":"740cda97-9bb7-46fe-a911-5dab641fa269","Type":"ContainerDied","Data":"7698177bb8309e81be92a3daf7a8e0c85d843655d9c035b8dde4204ff6e12d03"} Jul 17 12:45:08.472753 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.472409 2568 scope.go:117] "RemoveContainer" containerID="fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8" Jul 17 12:45:08.481040 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.481019 2568 scope.go:117] "RemoveContainer" containerID="fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8" Jul 17 12:45:08.481263 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:45:08.481247 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8\": container with ID starting with fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8 not found: ID does not exist" containerID="fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8" Jul 17 12:45:08.481323 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.481269 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8"} err="failed to get container status \"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8\": rpc error: code = NotFound desc = could not find container \"fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8\": container with ID starting with fb484293092d8ce5cf77f4b0aa8f072555837f21022966912740bf1ec388acc8 not found: ID does not exist" Jul 17 12:45:08.491919 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.491897 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:08.495020 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.495002 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7vqt"] Jul 17 12:45:08.588703 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:08.588653 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740cda97-9bb7-46fe-a911-5dab641fa269" path="/var/lib/kubelet/pods/740cda97-9bb7-46fe-a911-5dab641fa269/volumes" Jul 17 12:45:56.670081 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.670047 2568 generic.go:356] "Generic (PLEG): container finished" podID="b4809646-3859-4172-ae02-d2ce581c3c10" containerID="c885e864b93a63c8730844ab05189c9a5b8be95c2749f4a2216c7d8ff23dd1d9" exitCode=0 Jul 17 12:45:56.670469 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.670108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" event={"ID":"b4809646-3859-4172-ae02-d2ce581c3c10","Type":"ContainerDied","Data":"c885e864b93a63c8730844ab05189c9a5b8be95c2749f4a2216c7d8ff23dd1d9"} Jul 17 12:45:56.670469 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.670459 2568 scope.go:117] "RemoveContainer" containerID="c885e864b93a63c8730844ab05189c9a5b8be95c2749f4a2216c7d8ff23dd1d9" Jul 17 12:45:56.671481 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.671453 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:45:56.671538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.671528 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:45:56.671571 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:56.671558 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 12:45:57.675831 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:57.675799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" event={"ID":"b4809646-3859-4172-ae02-d2ce581c3c10","Type":"ContainerStarted","Data":"c466fc48823c032813793da10b8be0abd737c77aa56abe646f9d1bcef39f153c"} Jul 17 12:45:58.335704 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:58.335649 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:45:58.336673 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:58.336646 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gqxlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 17 12:45:58.336787 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:58.336723 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" podUID="b4809646-3859-4172-ae02-d2ce581c3c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 17 12:45:59.335384 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:59.335340 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gqxlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 17 12:45:59.335784 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:45:59.335407 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" podUID="b4809646-3859-4172-ae02-d2ce581c3c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 17 12:46:00.335425 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:00.335387 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gqxlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 17 12:46:00.335807 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:00.335443 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" podUID="b4809646-3859-4172-ae02-d2ce581c3c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 17 12:46:01.307173 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:01.307126 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:46:01.335213 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:01.335182 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-gqxlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 17 12:46:01.335361 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:01.335244 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" podUID="b4809646-3859-4172-ae02-d2ce581c3c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 17 12:46:02.338343 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:02.338312 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:46:02.339294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:02.339274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-gqxlk" Jul 17 12:46:53.558466 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.558435 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw"] Jul 17 12:46:53.558871 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.558814 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740cda97-9bb7-46fe-a911-5dab641fa269" containerName="authorino" Jul 17 12:46:53.558871 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.558825 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="740cda97-9bb7-46fe-a911-5dab641fa269" containerName="authorino" Jul 17 12:46:53.558941 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.558893 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="740cda97-9bb7-46fe-a911-5dab641fa269" containerName="authorino" Jul 17 12:46:53.561182 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.561164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.564467 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.564443 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 12:46:53.564646 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.564475 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdpj4\"" Jul 17 12:46:53.564774 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.564480 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 12:46:53.564774 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.564473 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 12:46:53.573063 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.573040 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw"] Jul 17 12:46:53.627289 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.627442 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2pfb\" (UniqueName: \"kubernetes.io/projected/f8970582-873c-4c5f-8507-94bd61961614-kube-api-access-r2pfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.627442 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8970582-873c-4c5f-8507-94bd61961614-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.627442 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627399 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.627442 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.627633 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.627452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728058 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2pfb\" (UniqueName: \"kubernetes.io/projected/f8970582-873c-4c5f-8507-94bd61961614-kube-api-access-r2pfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728058 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8970582-873c-4c5f-8507-94bd61961614-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728294 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728482 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728440 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728568 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728520 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.728709 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.728542 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.730538 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.730512 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8970582-873c-4c5f-8507-94bd61961614-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.730761 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.730743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8970582-873c-4c5f-8507-94bd61961614-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.737835 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.737816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2pfb\" (UniqueName: \"kubernetes.io/projected/f8970582-873c-4c5f-8507-94bd61961614-kube-api-access-r2pfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-l7wxw\" (UID: \"f8970582-873c-4c5f-8507-94bd61961614\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:53.872842 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:53.872748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:46:54.001975 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:54.001952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw"] Jul 17 12:46:54.004484 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:46:54.004451 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8970582_873c_4c5f_8507_94bd61961614.slice/crio-786e705a4ef9d63634d68a8f3bce4db81f9965355c2c4a937a4a79004a082bf2 WatchSource:0}: Error finding container 786e705a4ef9d63634d68a8f3bce4db81f9965355c2c4a937a4a79004a082bf2: Status 404 returned error can't find the container with id 786e705a4ef9d63634d68a8f3bce4db81f9965355c2c4a937a4a79004a082bf2 Jul 17 12:46:54.005843 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:54.005826 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 12:46:54.923168 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:54.923122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" event={"ID":"f8970582-873c-4c5f-8507-94bd61961614","Type":"ContainerStarted","Data":"786e705a4ef9d63634d68a8f3bce4db81f9965355c2c4a937a4a79004a082bf2"} Jul 17 12:46:55.929055 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:55.929013 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" event={"ID":"f8970582-873c-4c5f-8507-94bd61961614","Type":"ContainerStarted","Data":"e1fb62bae62220474283216b2f8884a1cbda10f5ca9922ece0dc2130f31bb339"} Jul 17 12:46:55.945486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:46:55.945438 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" podStartSLOduration=1.881309188 podStartE2EDuration="2.945423163s" totalImagesPullingTime="1.064113975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:46:55.943678094 +0000 UTC m=+1465.888537598" watchObservedRunningTime="2026-07-17 12:46:55.945423163 +0000 UTC m=+1465.890282702" Jul 17 12:47:02.868932 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:02.868900 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp"] Jul 17 12:47:02.871183 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:02.871167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:02.873486 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:02.873462 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 12:47:02.885468 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:02.885447 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp"] Jul 17 12:47:03.015945 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.015914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzcr\" (UniqueName: \"kubernetes.io/projected/8eb299a6-2e05-435b-9b1b-503c4cc9381e-kube-api-access-5mzcr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.016140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.015952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.016140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.015973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.016140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.016067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.016306 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.016164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.016306 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.016220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117481 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117659 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzcr\" (UniqueName: \"kubernetes.io/projected/8eb299a6-2e05-435b-9b1b-503c4cc9381e-kube-api-access-5mzcr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117659 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117659 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117659 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117916 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.117916 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.118002 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117927 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.118002 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.117966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.119949 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.119895 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb299a6-2e05-435b-9b1b-503c4cc9381e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.120164 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.120146 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb299a6-2e05-435b-9b1b-503c4cc9381e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.127037 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.127019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzcr\" (UniqueName: \"kubernetes.io/projected/8eb299a6-2e05-435b-9b1b-503c4cc9381e-kube-api-access-5mzcr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8fddp\" (UID: \"8eb299a6-2e05-435b-9b1b-503c4cc9381e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.180047 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.180008 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:03.307850 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.307829 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp"] Jul 17 12:47:03.309889 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:47:03.309862 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb299a6_2e05_435b_9b1b_503c4cc9381e.slice/crio-6b297c7fc4aee28b4ce9c150395c3ba75a101614c5854cdb4e084acbac0cb9fd WatchSource:0}: Error finding container 6b297c7fc4aee28b4ce9c150395c3ba75a101614c5854cdb4e084acbac0cb9fd: Status 404 returned error can't find the container with id 6b297c7fc4aee28b4ce9c150395c3ba75a101614c5854cdb4e084acbac0cb9fd Jul 17 12:47:03.873077 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.872973 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:47:03.873077 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.873054 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:47:03.885611 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.885582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:47:03.965884 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.965839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" event={"ID":"8eb299a6-2e05-435b-9b1b-503c4cc9381e","Type":"ContainerStarted","Data":"c6a41f46e8a78b6c9508c856f70ca36fd25ab2ac10574f262c03ca79d2b83ee0"} Jul 17 12:47:03.966056 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.965909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" event={"ID":"8eb299a6-2e05-435b-9b1b-503c4cc9381e","Type":"ContainerStarted","Data":"6b297c7fc4aee28b4ce9c150395c3ba75a101614c5854cdb4e084acbac0cb9fd"} Jul 17 12:47:03.977070 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:03.977039 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-l7wxw" Jul 17 12:47:04.015490 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:04.015436 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" podStartSLOduration=1.776157059 podStartE2EDuration="2.01542177s" totalImagesPullingTime="239.264711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:47:04.013571675 +0000 UTC m=+1473.958431177" watchObservedRunningTime="2026-07-17 12:47:04.01542177 +0000 UTC m=+1473.960281272" Jul 17 12:47:13.180498 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:13.180446 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:13.180974 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:13.180533 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:13.192791 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:13.192768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:47:14.023774 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:47:14.023746 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8fddp" Jul 17 12:48:32.890941 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:32.890910 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-778dd9b68-kvf29"] Jul 17 12:48:32.893572 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:32.893556 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:32.913197 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:32.913176 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-778dd9b68-kvf29"] Jul 17 12:48:32.993946 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:32.993917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98143c94-1355-49e8-9e17-34a18e746a62-tls-cert\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:32.994112 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:32.994015 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7b98\" (UniqueName: \"kubernetes.io/projected/98143c94-1355-49e8-9e17-34a18e746a62-kube-api-access-m7b98\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.095140 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.095090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7b98\" (UniqueName: \"kubernetes.io/projected/98143c94-1355-49e8-9e17-34a18e746a62-kube-api-access-m7b98\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.095309 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.095156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98143c94-1355-49e8-9e17-34a18e746a62-tls-cert\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.097606 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.097586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98143c94-1355-49e8-9e17-34a18e746a62-tls-cert\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.103966 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.103948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7b98\" (UniqueName: \"kubernetes.io/projected/98143c94-1355-49e8-9e17-34a18e746a62-kube-api-access-m7b98\") pod \"authorino-778dd9b68-kvf29\" (UID: \"98143c94-1355-49e8-9e17-34a18e746a62\") " pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.201987 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.201952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-778dd9b68-kvf29" Jul 17 12:48:33.335844 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:33.335806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-778dd9b68-kvf29"] Jul 17 12:48:33.339791 ip-10-0-129-112 kubenswrapper[2568]: W0717 12:48:33.339761 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98143c94_1355_49e8_9e17_34a18e746a62.slice/crio-800bd2f9853ee37846f9a92f120cbb5a08e0d499f115a240e21f9aa17b6edf2f WatchSource:0}: Error finding container 800bd2f9853ee37846f9a92f120cbb5a08e0d499f115a240e21f9aa17b6edf2f: Status 404 returned error can't find the container with id 800bd2f9853ee37846f9a92f120cbb5a08e0d499f115a240e21f9aa17b6edf2f Jul 17 12:48:34.348080 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.348037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-778dd9b68-kvf29" event={"ID":"98143c94-1355-49e8-9e17-34a18e746a62","Type":"ContainerStarted","Data":"12613f60a352903f31a7a221ac6c1854ba9efbfde439393b90cec020dfada99c"} Jul 17 12:48:34.348457 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.348092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-778dd9b68-kvf29" event={"ID":"98143c94-1355-49e8-9e17-34a18e746a62","Type":"ContainerStarted","Data":"800bd2f9853ee37846f9a92f120cbb5a08e0d499f115a240e21f9aa17b6edf2f"} Jul 17 12:48:34.372867 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.372798 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-778dd9b68-kvf29" podStartSLOduration=1.924476518 podStartE2EDuration="2.372782718s" totalImagesPullingTime="448.3062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:48:34.372138511 +0000 UTC m=+1564.316998016" watchObservedRunningTime="2026-07-17 12:48:34.372782718 +0000 UTC m=+1564.317642221" Jul 17 12:48:34.415396 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.415299 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:48:34.415597 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.415468 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f9884949-8k59k" podUID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" containerName="authorino" containerID="cri-o://236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4" gracePeriod=30 Jul 17 12:48:34.684511 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.684488 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:48:34.812675 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.812637 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5hn6\" (UniqueName: \"kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6\") pod \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " Jul 17 12:48:34.812856 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.812779 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert\") pod \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\" (UID: \"486b5ace-7d2c-4c00-8f44-45b9326b15f9\") " Jul 17 12:48:34.822414 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.821832 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6" (OuterVolumeSpecName: "kube-api-access-v5hn6") pod "486b5ace-7d2c-4c00-8f44-45b9326b15f9" (UID: "486b5ace-7d2c-4c00-8f44-45b9326b15f9"). InnerVolumeSpecName "kube-api-access-v5hn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 12:48:34.833657 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.833626 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "486b5ace-7d2c-4c00-8f44-45b9326b15f9" (UID: "486b5ace-7d2c-4c00-8f44-45b9326b15f9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 12:48:34.914328 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.914245 2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/486b5ace-7d2c-4c00-8f44-45b9326b15f9-tls-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:48:34.914328 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:34.914272 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5hn6\" (UniqueName: \"kubernetes.io/projected/486b5ace-7d2c-4c00-8f44-45b9326b15f9-kube-api-access-v5hn6\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 12:48:35.353672 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.353574 2568 generic.go:356] "Generic (PLEG): container finished" podID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" containerID="236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4" exitCode=0 Jul 17 12:48:35.353672 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.353623 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f9884949-8k59k" Jul 17 12:48:35.353672 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.353658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f9884949-8k59k" event={"ID":"486b5ace-7d2c-4c00-8f44-45b9326b15f9","Type":"ContainerDied","Data":"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4"} Jul 17 12:48:35.354232 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.353722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f9884949-8k59k" event={"ID":"486b5ace-7d2c-4c00-8f44-45b9326b15f9","Type":"ContainerDied","Data":"d230096613b70c90d40a6627c6801650325a968ec35175c93723127a581f98dc"} Jul 17 12:48:35.354232 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.353743 2568 scope.go:117] "RemoveContainer" containerID="236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4" Jul 17 12:48:35.362810 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.362777 2568 scope.go:117] "RemoveContainer" containerID="236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4" Jul 17 12:48:35.363118 ip-10-0-129-112 kubenswrapper[2568]: E0717 12:48:35.363087 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4\": container with ID starting with 236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4 not found: ID does not exist" containerID="236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4" Jul 17 12:48:35.363172 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.363131 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4"} err="failed to get container status \"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4\": rpc error: code = NotFound desc = could not find container \"236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4\": container with ID starting with 236828ea31b7302897949d4350a1036a42c7bbf3832d4cf5e74a2d104c2dafd4 not found: ID does not exist" Jul 17 12:48:35.398148 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.398114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:48:35.408018 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:35.407989 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f9884949-8k59k"] Jul 17 12:48:36.586321 ip-10-0-129-112 kubenswrapper[2568]: I0717 12:48:36.586287 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" path="/var/lib/kubelet/pods/486b5ace-7d2c-4c00-8f44-45b9326b15f9/volumes" Jul 17 13:02:35.017173 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.017137 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:02:35.017807 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.017639 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" containerName="authorino" Jul 17 13:02:35.017807 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.017660 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" containerName="authorino" Jul 17 13:02:35.017807 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.017798 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="486b5ace-7d2c-4c00-8f44-45b9326b15f9" containerName="authorino" Jul 17 13:02:35.020815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.020794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.024293 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.024275 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3fc8989b-openshift-default-dockercfg-gzq6b\"" Jul 17 13:02:35.044121 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.044086 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:02:35.180675 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180675 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180705 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180925 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180743 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180925 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180925 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnpqg\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180925 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.180925 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.181138 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.180933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.181138 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.181027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282493 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282493 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnpqg\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282565 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282565 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282565 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282737 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282737 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282737 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.282973 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.283054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.283054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.282996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.283363 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.283343 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.285021 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.285004 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.285155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.285135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.292136 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.292115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnpqg\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.292531 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.292504 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token\") pod \"e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.330762 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.330735 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:35.471278 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.471254 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:02:35.473823 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:02:35.473797 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8585867c_3b42_4a1d_a2a9_40cd1694bf37.slice/crio-f6f18f214c0dfc95326862914be90b06fa7dda416c22a73dc0ebe426dd8f370e WatchSource:0}: Error finding container f6f18f214c0dfc95326862914be90b06fa7dda416c22a73dc0ebe426dd8f370e: Status 404 returned error can't find the container with id f6f18f214c0dfc95326862914be90b06fa7dda416c22a73dc0ebe426dd8f370e Jul 17 13:02:35.475957 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.475923 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:35.476031 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.476005 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:35.476072 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.476032 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:35.940645 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.940612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerStarted","Data":"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1"} Jul 17 13:02:35.940817 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.940659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerStarted","Data":"f6f18f214c0dfc95326862914be90b06fa7dda416c22a73dc0ebe426dd8f370e"} Jul 17 13:02:35.962441 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:35.962390 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podStartSLOduration=1.962374466 podStartE2EDuration="1.962374466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:02:35.96090823 +0000 UTC m=+2405.905767733" watchObservedRunningTime="2026-07-17 13:02:35.962374466 +0000 UTC m=+2405.907233969" Jul 17 13:02:36.331960 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:36.331878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:36.337121 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:36.337096 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:36.945343 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:36.945319 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:36.946425 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:36.946404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:46.989253 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.989217 2568 generic.go:356] "Generic (PLEG): container finished" podID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerID="75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1" exitCode=0 Jul 17 13:02:46.989659 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.989263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerDied","Data":"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1"} Jul 17 13:02:46.989659 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.989627 2568 scope.go:117] "RemoveContainer" containerID="75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1" Jul 17 13:02:46.990722 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.990670 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:46.990787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.990761 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:46.990822 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:46.990792 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:02:47.999255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:47.999206 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerStarted","Data":"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1"} Jul 17 13:02:48.332020 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:48.331928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:48.333147 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:48.333123 2568 patch_prober.go:28] interesting pod/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 17 13:02:48.333219 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:48.333200 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 17 13:02:49.332073 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:49.332031 2568 patch_prober.go:28] interesting pod/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 17 13:02:49.332474 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:49.332102 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 17 13:02:50.331408 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:50.331364 2568 patch_prober.go:28] interesting pod/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 17 13:02:50.331601 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:50.331464 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 17 13:02:51.331826 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:51.331786 2568 patch_prober.go:28] interesting pod/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 17 13:02:51.332256 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:51.331857 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 17 13:02:51.945663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:51.945631 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:52.335439 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:52.335361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:02:52.336460 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:02:52.336439 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:05:41.389379 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:41.389335 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:05:41.389926 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:41.389540 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" containerID="cri-o://a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1" gracePeriod=30 Jul 17 13:05:46.628639 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.628605 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:05:46.781990 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.781959 2568 generic.go:356] "Generic (PLEG): container finished" podID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerID="a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1" exitCode=0 Jul 17 13:05:46.782143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.782011 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" Jul 17 13:05:46.782143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.782047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerDied","Data":"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1"} Jul 17 13:05:46.782143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.782105 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd" event={"ID":"8585867c-3b42-4a1d-a2a9-40cd1694bf37","Type":"ContainerDied","Data":"f6f18f214c0dfc95326862914be90b06fa7dda416c22a73dc0ebe426dd8f370e"} Jul 17 13:05:46.782143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.782134 2568 scope.go:117] "RemoveContainer" containerID="a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1" Jul 17 13:05:46.790093 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790077 2568 scope.go:117] "RemoveContainer" containerID="75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1" Jul 17 13:05:46.790597 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790583 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790706 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790672 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790748 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790780 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790815 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790917 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790838 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.790917 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790877 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.791068 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.790956 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.791068 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791035 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnpqg\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg\") pod \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\" (UID: \"8585867c-3b42-4a1d-a2a9-40cd1694bf37\") " Jul 17 13:05:46.791420 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791393 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data" (OuterVolumeSpecName: "istio-data") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:05:46.791567 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791417 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:05:46.791567 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791500 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:05:46.791814 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791789 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:05:46.791894 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.791810 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:05:46.793387 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.793368 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:05:46.793847 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.793822 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg" (OuterVolumeSpecName: "kube-api-access-jnpqg") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "kube-api-access-jnpqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:05:46.794025 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.794007 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:05:46.794089 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.794026 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token" (OuterVolumeSpecName: "istio-token") pod "8585867c-3b42-4a1d-a2a9-40cd1694bf37" (UID: "8585867c-3b42-4a1d-a2a9-40cd1694bf37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:05:46.809286 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.809268 2568 scope.go:117] "RemoveContainer" containerID="a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1" Jul 17 13:05:46.809554 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:05:46.809535 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1\": container with ID starting with a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1 not found: ID does not exist" containerID="a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1" Jul 17 13:05:46.809627 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.809565 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1"} err="failed to get container status \"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1\": rpc error: code = NotFound desc = could not find container \"a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1\": container with ID starting with a3b539d10311a26ec7940c742ab6096e50a27ab9bda3d26f7326445e8d2c0bb1 not found: ID does not exist" Jul 17 13:05:46.809627 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.809590 2568 scope.go:117] "RemoveContainer" containerID="75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1" Jul 17 13:05:46.809863 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:05:46.809848 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1\": container with ID starting with 75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1 not found: ID does not exist" containerID="75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1" Jul 17 13:05:46.809915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.809869 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1"} err="failed to get container status \"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1\": rpc error: code = NotFound desc = could not find container \"75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1\": container with ID starting with 75c5cadb90961536d8126ee7a877f05e2924eab9e49114c3bce8e74df8050ef1 not found: ID does not exist" Jul 17 13:05:46.892018 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.891978 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnpqg\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-kube-api-access-jnpqg\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892018 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892013 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892018 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892023 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892018 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892032 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892041 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892048 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892057 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892064 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:46.892238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:46.892072 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8585867c-3b42-4a1d-a2a9-40cd1694bf37-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:05:47.105484 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:47.105453 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:05:47.108909 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:47.108885 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3fc8989b-openshift-default-6757cc9879-g8fnd"] Jul 17 13:05:48.592424 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:48.592388 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" path="/var/lib/kubelet/pods/8585867c-3b42-4a1d-a2a9-40cd1694bf37/volumes" Jul 17 13:05:54.456264 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.456190 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:05:54.456671 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.456655 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:05:54.456768 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.456675 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:05:54.456800 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.456781 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:05:54.456800 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.456795 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:05:54.461354 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.461335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.463830 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.463811 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-359d73d8-gw-openshift-default-dockercfg-x6tln\"" Jul 17 13:05:54.481837 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.481813 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:05:54.553739 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.553739 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.553955 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553777 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.553955 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.553955 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.553955 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553894 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgjz\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.554146 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.553963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.554146 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.554032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.554146 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.554059 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655443 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655443 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655488 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgjz\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.655677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.656016 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.655974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.656207 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.656180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.656296 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.656208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.656418 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.656396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.656476 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.656427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.657923 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.657905 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.658081 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.658064 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.666264 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.666243 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgjz\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.666343 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.666287 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token\") pod \"e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.771647 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.771567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:54.909254 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.909229 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:05:54.910928 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:05:54.910900 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d2ee8_4d85_4793_b913_15df2104c8ed.slice/crio-d63224610ce61bfac7ddd7c76c89e7090428f707f4c0f53759204998a76c2321 WatchSource:0}: Error finding container d63224610ce61bfac7ddd7c76c89e7090428f707f4c0f53759204998a76c2321: Status 404 returned error can't find the container with id d63224610ce61bfac7ddd7c76c89e7090428f707f4c0f53759204998a76c2321 Jul 17 13:05:54.912929 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.912897 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:05:54.913008 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.912957 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:05:54.913008 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:54.912985 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:05:55.824244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:55.824209 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerStarted","Data":"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b"} Jul 17 13:05:55.824732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:55.824260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerStarted","Data":"d63224610ce61bfac7ddd7c76c89e7090428f707f4c0f53759204998a76c2321"} Jul 17 13:05:55.847943 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:55.847898 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" podStartSLOduration=1.847883655 podStartE2EDuration="1.847883655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:05:55.844998586 +0000 UTC m=+2605.789858088" watchObservedRunningTime="2026-07-17 13:05:55.847883655 +0000 UTC m=+2605.792743158" Jul 17 13:05:56.772060 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:56.772027 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:56.776983 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:56.776959 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:56.830313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:56.830284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:05:56.831186 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:05:56.831168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:06:06.877211 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.877181 2568 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerID="29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b" exitCode=0 Jul 17 13:06:06.877606 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.877234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerDied","Data":"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b"} Jul 17 13:06:06.877606 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.877598 2568 scope.go:117] "RemoveContainer" containerID="29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b" Jul 17 13:06:06.878560 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.878528 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:06:06.878657 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.878597 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:06:06.878657 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:06.878627 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:06:07.884930 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:07.884223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerStarted","Data":"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04"} Jul 17 13:06:08.772588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:08.772547 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:06:08.773696 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:08.773658 2568 patch_prober.go:28] interesting pod/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 17 13:06:08.773814 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:08.773740 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 17 13:06:09.772131 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:09.772090 2568 patch_prober.go:28] interesting pod/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 17 13:06:09.772596 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:09.772171 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 17 13:06:10.772166 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:10.772132 2568 patch_prober.go:28] interesting pod/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 17 13:06:10.772551 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:10.772187 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 17 13:06:11.775903 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:11.775872 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:06:11.776305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:11.776137 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:06:11.776745 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:06:11.776730 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:07:00.350786 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:00.350750 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:07:00.351192 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:00.350986 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" containerID="cri-o://2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04" gracePeriod=30 Jul 17 13:07:05.604838 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.604812 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:07:05.687858 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.687816 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.687899 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.687927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stgjz\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.687970 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688070 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688094 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688123 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688163 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688193 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket\") pod \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\" (UID: \"9b8d2ee8-4d85-4793-b913-15df2104c8ed\") " Jul 17 13:07:05.688439 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688259 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:07:05.688439 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688384 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:07:05.688538 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688483 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:07:05.688538 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688528 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.688823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688550 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.688823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.688607 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:07:05.689615 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.689586 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:07:05.690310 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.690285 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:07:05.690416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.690397 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz" (OuterVolumeSpecName: "kube-api-access-stgjz") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "kube-api-access-stgjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:07:05.690601 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.690581 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:07:05.690834 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.690818 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b8d2ee8-4d85-4793-b913-15df2104c8ed" (UID: "9b8d2ee8-4d85-4793-b913-15df2104c8ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:07:05.789305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789261 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789296 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789306 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789315 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789559 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789323 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789559 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789332 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stgjz\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-kube-api-access-stgjz\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:05.789559 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:05.789339 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8d2ee8-4d85-4793-b913-15df2104c8ed-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:07:06.149842 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.149814 2568 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerID="2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04" exitCode=0 Jul 17 13:07:06.150010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.149870 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" Jul 17 13:07:06.150010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.149898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerDied","Data":"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04"} Jul 17 13:07:06.150010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.149955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz" event={"ID":"9b8d2ee8-4d85-4793-b913-15df2104c8ed","Type":"ContainerDied","Data":"d63224610ce61bfac7ddd7c76c89e7090428f707f4c0f53759204998a76c2321"} Jul 17 13:07:06.150010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.149989 2568 scope.go:117] "RemoveContainer" containerID="2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04" Jul 17 13:07:06.158791 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.158761 2568 scope.go:117] "RemoveContainer" containerID="29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b" Jul 17 13:07:06.167105 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.167087 2568 scope.go:117] "RemoveContainer" containerID="2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04" Jul 17 13:07:06.167408 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:07:06.167388 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04\": container with ID starting with 2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04 not found: ID does not exist" containerID="2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04" Jul 17 13:07:06.167452 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.167420 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04"} err="failed to get container status \"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04\": rpc error: code = NotFound desc = could not find container \"2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04\": container with ID starting with 2ec3bc207014db8a76870c0131edaa258f72997769c215b8e3216d362b084e04 not found: ID does not exist" Jul 17 13:07:06.167452 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.167439 2568 scope.go:117] "RemoveContainer" containerID="29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b" Jul 17 13:07:06.167706 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:07:06.167668 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b\": container with ID starting with 29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b not found: ID does not exist" containerID="29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b" Jul 17 13:07:06.167757 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.167715 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b"} err="failed to get container status \"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b\": rpc error: code = NotFound desc = could not find container \"29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b\": container with ID starting with 29d5067e73edae6ebc347d916192885d3baecbfd042be0017c3c92163a56cb6b not found: ID does not exist" Jul 17 13:07:06.178145 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.178110 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:07:06.185860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.185832 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-359d73d8-gw-openshift-default-66bc5f8bc-q7gnz"] Jul 17 13:07:06.586883 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:06.586795 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" path="/var/lib/kubelet/pods/9b8d2ee8-4d85-4793-b913-15df2104c8ed/volumes" Jul 17 13:07:12.866134 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866101 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866408 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866418 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866435 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866440 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8585867c-3b42-4a1d-a2a9-40cd1694bf37" containerName="istio-proxy" Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866502 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:07:12.866514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.866512 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:07:12.868733 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.868716 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.871437 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.871414 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-12f6c9d7-openshift-default-dockercfg-wd2xp\"" Jul 17 13:07:12.885232 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.885202 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:07:12.960229 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960229 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960441 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf2g\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960441 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960441 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960441 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960564 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960564 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:12.960564 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:12.960560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061620 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061857 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061857 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061857 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061857 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.061857 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061860 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf2g\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.061988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062259 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.062143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062259 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.062170 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062259 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.062190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062491 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.062467 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.062593 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.062573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.064177 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.064142 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.064306 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.064290 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.072102 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.072080 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf2g\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.073713 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.073675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token\") pod \"e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.183032 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.182995 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:13.323536 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.323443 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:07:13.325862 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:07:13.325837 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f532758_0170_4c3a_9c8f_151279ca5855.slice/crio-b42c0fbb81776f0418b911efed34def3bd523fee69ec62d76d676a2c0b8ae840 WatchSource:0}: Error finding container b42c0fbb81776f0418b911efed34def3bd523fee69ec62d76d676a2c0b8ae840: Status 404 returned error can't find the container with id b42c0fbb81776f0418b911efed34def3bd523fee69ec62d76d676a2c0b8ae840 Jul 17 13:07:13.327888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.327848 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:13.327961 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.327922 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:13.328011 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:13.327958 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:14.186258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:14.186221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerStarted","Data":"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd"} Jul 17 13:07:14.186636 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:14.186280 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerStarted","Data":"b42c0fbb81776f0418b911efed34def3bd523fee69ec62d76d676a2c0b8ae840"} Jul 17 13:07:14.217744 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:14.214986 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" podStartSLOduration=2.214967082 podStartE2EDuration="2.214967082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:14.214605402 +0000 UTC m=+2684.159464906" watchObservedRunningTime="2026-07-17 13:07:14.214967082 +0000 UTC m=+2684.159826585" Jul 17 13:07:15.183823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:15.183784 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:15.188991 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:15.188963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:15.191018 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:15.190988 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:15.191923 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:15.191900 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:25.236982 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.236946 2568 generic.go:356] "Generic (PLEG): container finished" podID="7f532758-0170-4c3a-9c8f-151279ca5855" containerID="2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd" exitCode=0 Jul 17 13:07:25.237397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.237020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerDied","Data":"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd"} Jul 17 13:07:25.237397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.237376 2568 scope.go:117] "RemoveContainer" containerID="2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd" Jul 17 13:07:25.238348 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.238314 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:25.238501 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.238407 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:25.238501 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:25.238457 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:07:26.243475 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:26.243441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerStarted","Data":"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5"} Jul 17 13:07:27.184320 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:27.184266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:27.185437 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:27.185397 2568 patch_prober.go:28] interesting pod/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 17 13:07:27.185573 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:27.185463 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 17 13:07:28.183836 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:28.183795 2568 patch_prober.go:28] interesting pod/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 17 13:07:28.184319 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:28.183869 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 17 13:07:29.183625 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:29.183582 2568 patch_prober.go:28] interesting pod/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 17 13:07:29.183938 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:29.183903 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 17 13:07:30.188160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:30.188128 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:30.188591 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:30.188417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:07:30.189054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:07:30.189037 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:13:19.414019 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:19.413982 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:13:19.414600 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:19.414176 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" containerID="cri-o://a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5" gracePeriod=30 Jul 17 13:13:24.657483 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.657458 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:13:24.695618 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695618 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695588 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695620 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695651 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695744 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695777 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695805 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlf2g\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.695840 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695828 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.696119 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.695932 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data\") pod \"7f532758-0170-4c3a-9c8f-151279ca5855\" (UID: \"7f532758-0170-4c3a-9c8f-151279ca5855\") " Jul 17 13:13:24.697004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.696931 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:13:24.697004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.696946 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:13:24.697197 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.696867 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:13:24.697782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.697531 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:13:24.697782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.697735 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:13:24.699134 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.699087 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:13:24.699134 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.699100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g" (OuterVolumeSpecName: "kube-api-access-vlf2g") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "kube-api-access-vlf2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:13:24.699280 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.699228 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:13:24.699636 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.699609 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f532758-0170-4c3a-9c8f-151279ca5855" (UID: "7f532758-0170-4c3a-9c8f-151279ca5855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:13:24.783255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.783226 2568 generic.go:356] "Generic (PLEG): container finished" podID="7f532758-0170-4c3a-9c8f-151279ca5855" containerID="a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5" exitCode=0 Jul 17 13:13:24.783390 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.783269 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerDied","Data":"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5"} Jul 17 13:13:24.783390 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.783274 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" Jul 17 13:13:24.783390 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.783311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc" event={"ID":"7f532758-0170-4c3a-9c8f-151279ca5855","Type":"ContainerDied","Data":"b42c0fbb81776f0418b911efed34def3bd523fee69ec62d76d676a2c0b8ae840"} Jul 17 13:13:24.783390 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.783332 2568 scope.go:117] "RemoveContainer" containerID="a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5" Jul 17 13:13:24.793232 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.793211 2568 scope.go:117] "RemoveContainer" containerID="2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd" Jul 17 13:13:24.796981 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.796956 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf2g\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-kube-api-access-vlf2g\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.796984 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f532758-0170-4c3a-9c8f-151279ca5855-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.796997 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797008 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797019 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797030 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797041 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f532758-0170-4c3a-9c8f-151279ca5855-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797052 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f532758-0170-4c3a-9c8f-151279ca5855-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.797083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.797062 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f532758-0170-4c3a-9c8f-151279ca5855-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:13:24.802433 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.802416 2568 scope.go:117] "RemoveContainer" containerID="a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5" Jul 17 13:13:24.802700 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:13:24.802668 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5\": container with ID starting with a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5 not found: ID does not exist" containerID="a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5" Jul 17 13:13:24.802774 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.802714 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5"} err="failed to get container status \"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5\": rpc error: code = NotFound desc = could not find container \"a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5\": container with ID starting with a7b80c7f6c3c099ec3a8c516adb64c7cd110417d38a4a08ee8ec305fb9e6eec5 not found: ID does not exist" Jul 17 13:13:24.802774 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.802737 2568 scope.go:117] "RemoveContainer" containerID="2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd" Jul 17 13:13:24.802972 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:13:24.802950 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd\": container with ID starting with 2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd not found: ID does not exist" containerID="2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd" Jul 17 13:13:24.803022 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.802978 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd"} err="failed to get container status \"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd\": rpc error: code = NotFound desc = could not find container \"2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd\": container with ID starting with 2f4b76c8b9dc494cc3e623fcd660630e28ad9a4900800c4ac5846bad857ba2dd not found: ID does not exist" Jul 17 13:13:24.807140 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.807112 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:13:24.810850 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:24.810828 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-12f6c9d7-openshift-default-75c996c5-7dpfc"] Jul 17 13:13:26.586030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:26.585999 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" path="/var/lib/kubelet/pods/7f532758-0170-4c3a-9c8f-151279ca5855/volumes" Jul 17 13:13:32.563560 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.563530 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:13:32.563989 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.563973 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:13:32.564032 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.563990 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8d2ee8-4d85-4793-b913-15df2104c8ed" containerName="istio-proxy" Jul 17 13:13:32.564032 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.564010 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:13:32.564032 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.564018 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:13:32.564117 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.564079 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:13:32.564117 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.564088 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:13:32.568648 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.568629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.570795 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.570776 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-661ee8a2-openshift-default-dockercfg-tj6v5\"" Jul 17 13:13:32.595083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.595050 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:13:32.663573 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.663769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.663769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.663769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.663769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.663769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.664004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcg4\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.664004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.664004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.663908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764452 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764452 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcg4\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764555 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.764732 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.765082 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.764909 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.765082 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.765014 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.765082 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.765052 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.765175 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.765099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.765501 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.765480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.767037 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.767009 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.767152 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.767136 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.778245 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.778222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.778450 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.778427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcg4\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4\") pod \"e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:32.878932 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:32.878857 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:33.037075 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.036880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:13:33.037496 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:13:33.037473 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa59700_405d_41d8_b884_27cfe6991ec8.slice/crio-09bfdd34e43f3fe261dba3b4b519215bfeb2ee5624ce62911e428086adccb5f5 WatchSource:0}: Error finding container 09bfdd34e43f3fe261dba3b4b519215bfeb2ee5624ce62911e428086adccb5f5: Status 404 returned error can't find the container with id 09bfdd34e43f3fe261dba3b4b519215bfeb2ee5624ce62911e428086adccb5f5 Jul 17 13:13:33.040848 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.040806 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:33.040948 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.040885 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:33.040948 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.040921 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:33.824199 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.824158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerStarted","Data":"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300"} Jul 17 13:13:33.824585 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.824222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerStarted","Data":"09bfdd34e43f3fe261dba3b4b519215bfeb2ee5624ce62911e428086adccb5f5"} Jul 17 13:13:33.864661 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.864611 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" podStartSLOduration=1.86459639 podStartE2EDuration="1.86459639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:33.863350944 +0000 UTC m=+3063.808210442" watchObservedRunningTime="2026-07-17 13:13:33.86459639 +0000 UTC m=+3063.809455892" Jul 17 13:13:33.879825 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:33.879801 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:34.883984 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:34.883951 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:35.832283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:35.832250 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:35.833304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:35.833285 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:44.875866 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.875836 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerID="b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300" exitCode=0 Jul 17 13:13:44.876280 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.875894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerDied","Data":"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300"} Jul 17 13:13:44.876280 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.876248 2568 scope.go:117] "RemoveContainer" containerID="b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300" Jul 17 13:13:44.877370 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.877337 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:44.877467 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.877422 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:44.877526 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:44.877470 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:13:45.881945 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:45.881905 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerStarted","Data":"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56"} Jul 17 13:13:46.879865 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:46.879831 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:46.880841 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:46.880817 2568 patch_prober.go:28] interesting pod/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 17 13:13:46.880974 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:46.880870 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 17 13:13:47.879678 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:47.879645 2568 patch_prober.go:28] interesting pod/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 17 13:13:47.880082 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:47.879732 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 17 13:13:48.879367 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:48.879333 2568 patch_prober.go:28] interesting pod/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 17 13:13:48.879541 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:48.879400 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 17 13:13:49.883653 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:49.883621 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:49.884153 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:49.884052 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:13:49.884818 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:13:49.884801 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:16:38.311783 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:38.311698 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:16:38.312256 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:38.311919 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" containerID="cri-o://8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56" gracePeriod=30 Jul 17 13:16:43.562308 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.562282 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:16:43.610062 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.609981 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wcg4\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610062 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610042 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610299 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610165 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610299 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610207 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610299 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610271 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610446 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610302 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610446 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610325 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610446 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610387 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610446 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610412 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data\") pod \"7fa59700-405d-41d8-b884-27cfe6991ec8\" (UID: \"7fa59700-405d-41d8-b884-27cfe6991ec8\") " Jul 17 13:16:43.610631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610479 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:16:43.610631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610511 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:16:43.610631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610540 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:16:43.611036 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610794 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa59700-405d-41d8-b884-27cfe6991ec8-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.611036 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610816 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.611036 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.610831 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.611036 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.611001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:16:43.611519 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.611413 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:16:43.612961 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.612923 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4" (OuterVolumeSpecName: "kube-api-access-6wcg4") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "kube-api-access-6wcg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:16:43.613078 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.613054 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:16:43.613379 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.613352 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:16:43.613450 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.613419 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fa59700-405d-41d8-b884-27cfe6991ec8" (UID: "7fa59700-405d-41d8-b884-27cfe6991ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:16:43.671464 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.671431 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerID="8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56" exitCode=0 Jul 17 13:16:43.671588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.671487 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" Jul 17 13:16:43.671588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.671497 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerDied","Data":"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56"} Jul 17 13:16:43.671588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.671550 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85" event={"ID":"7fa59700-405d-41d8-b884-27cfe6991ec8","Type":"ContainerDied","Data":"09bfdd34e43f3fe261dba3b4b519215bfeb2ee5624ce62911e428086adccb5f5"} Jul 17 13:16:43.671588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.671579 2568 scope.go:117] "RemoveContainer" containerID="8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56" Jul 17 13:16:43.680515 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.680497 2568 scope.go:117] "RemoveContainer" containerID="b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300" Jul 17 13:16:43.687706 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.687669 2568 scope.go:117] "RemoveContainer" containerID="8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56" Jul 17 13:16:43.687981 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:16:43.687964 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56\": container with ID starting with 8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56 not found: ID does not exist" containerID="8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56" Jul 17 13:16:43.688020 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.687991 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56"} err="failed to get container status \"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56\": rpc error: code = NotFound desc = could not find container \"8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56\": container with ID starting with 8951155dae23f33d86415315f638175d75e0ddf793e0c0d7a8b29e8c9765ac56 not found: ID does not exist" Jul 17 13:16:43.688020 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.688007 2568 scope.go:117] "RemoveContainer" containerID="b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300" Jul 17 13:16:43.688229 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:16:43.688208 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300\": container with ID starting with b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300 not found: ID does not exist" containerID="b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300" Jul 17 13:16:43.688272 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.688236 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300"} err="failed to get container status \"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300\": rpc error: code = NotFound desc = could not find container \"b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300\": container with ID starting with b80059e4a1a9a1cd6f18514f497b7a6afc9fe4313bb06784d563ac19c4864300 not found: ID does not exist" Jul 17 13:16:43.691893 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.691866 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:16:43.696887 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.696867 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-661ee8a2-openshift-default-6b46ddcfcf-d8r85"] Jul 17 13:16:43.711863 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711840 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.711951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711868 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.711951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711881 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.711951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711894 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.711951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711905 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa59700-405d-41d8-b884-27cfe6991ec8-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:43.711951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:43.711916 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcg4\" (UniqueName: \"kubernetes.io/projected/7fa59700-405d-41d8-b884-27cfe6991ec8-kube-api-access-6wcg4\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:16:44.586996 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:16:44.586969 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" path="/var/lib/kubelet/pods/7fa59700-405d-41d8-b884-27cfe6991ec8/volumes" Jul 17 13:18:27.092541 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092509 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092853 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092864 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092880 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092886 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092901 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092907 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f532758-0170-4c3a-9c8f-151279ca5855" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092963 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.093064 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.092972 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa59700-405d-41d8-b884-27cfe6991ec8" containerName="istio-proxy" Jul 17 13:18:27.096009 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.095988 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.098429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.098406 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cae85d5e-openshift-default-dockercfg-cpfdm\"" Jul 17 13:18:27.110277 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.110245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:18:27.261117 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92nj\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261334 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.261566 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.261445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362293 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362293 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362293 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92nj\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362518 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362542 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362920 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362672 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.362979 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.362948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.363035 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.363022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.363081 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.363028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.363126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.363106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.365454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.365431 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.365802 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.365781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.371397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.371370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92nj\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.371479 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.371398 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token\") pod \"e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.406561 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.406519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:27.552433 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.552399 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:18:27.556398 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:18:27.556371 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcc7927_2bed_4a44_9e4a_6898218e5d44.slice/crio-7c75c6ac3f05817984fd48d049c1d6389905bb2232d49050135d1300fb8c9f51 WatchSource:0}: Error finding container 7c75c6ac3f05817984fd48d049c1d6389905bb2232d49050135d1300fb8c9f51: Status 404 returned error can't find the container with id 7c75c6ac3f05817984fd48d049c1d6389905bb2232d49050135d1300fb8c9f51 Jul 17 13:18:27.558588 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.558553 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:27.558654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.558625 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:27.558716 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:27.558653 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:28.113600 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:28.113564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerStarted","Data":"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5"} Jul 17 13:18:28.114021 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:28.113617 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerStarted","Data":"7c75c6ac3f05817984fd48d049c1d6389905bb2232d49050135d1300fb8c9f51"} Jul 17 13:18:28.142510 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:28.142416 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podStartSLOduration=1.142392791 podStartE2EDuration="1.142392791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:18:28.142233949 +0000 UTC m=+3358.087093453" watchObservedRunningTime="2026-07-17 13:18:28.142392791 +0000 UTC m=+3358.087252298" Jul 17 13:18:28.407210 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:28.407175 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:28.412157 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:28.412130 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:29.123359 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:29.120845 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:29.124210 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:29.124186 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:43.185528 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.185488 2568 generic.go:356] "Generic (PLEG): container finished" podID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerID="c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5" exitCode=0 Jul 17 13:18:43.186126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.185558 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerDied","Data":"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5"} Jul 17 13:18:43.186126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.185987 2568 scope.go:117] "RemoveContainer" containerID="c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5" Jul 17 13:18:43.187044 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.187016 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:43.187115 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.187079 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:43.187169 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:43.187114 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:18:44.121238 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:44.121203 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:44.195708 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:44.195653 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerStarted","Data":"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b"} Jul 17 13:18:44.406993 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:44.406942 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:44.408225 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:44.408201 2568 patch_prober.go:28] interesting pod/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 17 13:18:44.408312 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:44.408254 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 17 13:18:45.407457 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:45.407411 2568 patch_prober.go:28] interesting pod/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 17 13:18:45.407930 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:45.407473 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 17 13:18:46.407837 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:46.407799 2568 patch_prober.go:28] interesting pod/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 17 13:18:46.408231 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:46.407871 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 17 13:18:47.407999 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:47.407951 2568 patch_prober.go:28] interesting pod/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 17 13:18:47.408387 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:47.408008 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 17 13:18:48.411201 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:48.411165 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:48.411574 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:48.411427 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:18:48.412214 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:18:48.412198 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:19:07.677125 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:07.677092 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:19:07.677628 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:07.677293 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" containerID="cri-o://64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b" gracePeriod=30 Jul 17 13:19:12.921030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.921003 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:19:12.986434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986336 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986373 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986402 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986436 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986549 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986610 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n92nj\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986634 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986679 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986747 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data\") pod \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\" (UID: \"1bcc7927-2bed-4a44-9e4a-6898218e5d44\") " Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986768 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:19:12.986877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.986810 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987015 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987139 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987165 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987181 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:12.987249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.987194 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:12.988274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.988245 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data" (OuterVolumeSpecName: "istio-data") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:19:12.988860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.988830 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:19:12.988860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.988841 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token" (OuterVolumeSpecName: "istio-token") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:19:12.989016 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.988997 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:19:12.989117 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:12.989100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj" (OuterVolumeSpecName: "kube-api-access-n92nj") pod "1bcc7927-2bed-4a44-9e4a-6898218e5d44" (UID: "1bcc7927-2bed-4a44-9e4a-6898218e5d44"). InnerVolumeSpecName "kube-api-access-n92nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:19:13.087667 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.087612 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n92nj\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-kube-api-access-n92nj\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:13.087667 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.087663 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:13.087667 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.087675 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:13.087948 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.087715 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:13.087948 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.087723 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc7927-2bed-4a44-9e4a-6898218e5d44-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:19:13.328287 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.328197 2568 generic.go:356] "Generic (PLEG): container finished" podID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerID="64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b" exitCode=0 Jul 17 13:19:13.328287 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.328242 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" Jul 17 13:19:13.328498 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.328288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerDied","Data":"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b"} Jul 17 13:19:13.328498 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.328349 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt" event={"ID":"1bcc7927-2bed-4a44-9e4a-6898218e5d44","Type":"ContainerDied","Data":"7c75c6ac3f05817984fd48d049c1d6389905bb2232d49050135d1300fb8c9f51"} Jul 17 13:19:13.328498 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.328373 2568 scope.go:117] "RemoveContainer" containerID="64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b" Jul 17 13:19:13.337834 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.337809 2568 scope.go:117] "RemoveContainer" containerID="c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5" Jul 17 13:19:13.345820 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.345801 2568 scope.go:117] "RemoveContainer" containerID="64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b" Jul 17 13:19:13.346090 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:19:13.346070 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b\": container with ID starting with 64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b not found: ID does not exist" containerID="64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b" Jul 17 13:19:13.346159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.346098 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b"} err="failed to get container status \"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b\": rpc error: code = NotFound desc = could not find container \"64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b\": container with ID starting with 64baca9de7f7fa47423a594bde6c1008f898f2a10c6d4812de973fbdb271e92b not found: ID does not exist" Jul 17 13:19:13.346159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.346116 2568 scope.go:117] "RemoveContainer" containerID="c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5" Jul 17 13:19:13.346364 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:19:13.346346 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5\": container with ID starting with c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5 not found: ID does not exist" containerID="c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5" Jul 17 13:19:13.346425 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.346370 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5"} err="failed to get container status \"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5\": rpc error: code = NotFound desc = could not find container \"c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5\": container with ID starting with c8cf3d2f29c9d1cd4fb730462ed6d5262a14fb242dfde39105ed30a24d00f5b5 not found: ID does not exist" Jul 17 13:19:13.350659 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.350635 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:19:13.355040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:13.355015 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cae85d5e-openshift-default-846c44fb5f-kzjxt"] Jul 17 13:19:14.587001 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:14.586968 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" path="/var/lib/kubelet/pods/1bcc7927-2bed-4a44-9e4a-6898218e5d44/volumes" Jul 17 13:19:28.601488 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.601406 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:19:28.601943 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.601921 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.601991 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.601948 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.601991 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.601967 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.601991 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.601976 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.602079 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.602057 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.602079 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.602071 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bcc7927-2bed-4a44-9e4a-6898218e5d44" containerName="istio-proxy" Jul 17 13:19:28.606872 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.606853 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.613130 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.613106 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6ad1d9a1-openshift-default-dockercfg-p2hr6\"" Jul 17 13:19:28.639470 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.639439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:19:28.713726 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.713921 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.713921 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713842 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.713921 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.713921 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713896 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.714428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.714428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.713982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.714428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.714029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.714428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.714054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9sw\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815535 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815535 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.815823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816067 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9sw\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816067 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816067 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815940 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816067 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816067 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.815985 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816314 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.816232 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.816635 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.816174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.823382 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.817145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.823661 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.823630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.823896 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.823874 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.829938 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.829915 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.830155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.830133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9sw\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw\") pod \"e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:28.916766 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:28.916721 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:29.065997 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.065962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:19:29.069108 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:19:29.069074 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b929ff_2486_4e63_8490_ba67d4328b1c.slice/crio-7bbbf61dc1a16123ac03b353cc4e3ab9fd50496f49cc9ff96c4da5f6c7d88f98 WatchSource:0}: Error finding container 7bbbf61dc1a16123ac03b353cc4e3ab9fd50496f49cc9ff96c4da5f6c7d88f98: Status 404 returned error can't find the container with id 7bbbf61dc1a16123ac03b353cc4e3ab9fd50496f49cc9ff96c4da5f6c7d88f98 Jul 17 13:19:29.071271 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.071234 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:29.071343 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.071308 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:29.071389 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.071344 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:29.400188 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.400151 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerStarted","Data":"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc"} Jul 17 13:19:29.400366 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.400204 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerStarted","Data":"7bbbf61dc1a16123ac03b353cc4e3ab9fd50496f49cc9ff96c4da5f6c7d88f98"} Jul 17 13:19:29.433602 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.433541 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podStartSLOduration=1.4335184650000001 podStartE2EDuration="1.433518465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:19:29.433350338 +0000 UTC m=+3419.378209842" watchObservedRunningTime="2026-07-17 13:19:29.433518465 +0000 UTC m=+3419.378377970" Jul 17 13:19:29.917749 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:29.917714 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:30.923179 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:30.923150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:31.408723 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:31.408693 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:31.409586 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:31.409568 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:45.471057 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.471024 2568 generic.go:356] "Generic (PLEG): container finished" podID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerID="fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc" exitCode=0 Jul 17 13:19:45.471475 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.471076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerDied","Data":"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc"} Jul 17 13:19:45.471475 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.471441 2568 scope.go:117] "RemoveContainer" containerID="fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc" Jul 17 13:19:45.472435 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.472403 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:45.472536 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.472485 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:45.472598 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:45.472534 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:19:46.408956 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:46.408905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:46.477100 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:46.477067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerStarted","Data":"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad"} Jul 17 13:19:46.918495 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:46.918454 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:46.919778 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:46.919751 2568 patch_prober.go:28] interesting pod/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 17 13:19:46.919900 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:46.919803 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 17 13:19:47.918346 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:47.918309 2568 patch_prober.go:28] interesting pod/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 17 13:19:47.918777 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:47.918366 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 17 13:19:48.917909 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:48.917864 2568 patch_prober.go:28] interesting pod/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 17 13:19:48.918126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:48.917916 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 17 13:19:49.918659 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:49.918610 2568 patch_prober.go:28] interesting pod/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 17 13:19:49.919062 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:49.918722 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 17 13:19:50.922044 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:50.922011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:50.922470 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:50.922368 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:19:50.923098 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:19:50.923080 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:20:13.681411 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.681374 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:20:13.683877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.683861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.687467 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.687444 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6ad1d9a1\"/\"default-dockercfg-gfbhf\"" Jul 17 13:20:13.687467 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.687451 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6ad1d9a1\"/\"kube-root-ca.crt\"" Jul 17 13:20:13.687656 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.687487 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6ad1d9a1\"/\"openshift-service-ca.crt\"" Jul 17 13:20:13.687656 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.687455 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6ad1d9a1\"/\"e2e-lifecycle-model-6ad1d9a1-kserve-self-signed-certs\"" Jul 17 13:20:13.699831 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.699803 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:20:13.748478 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6sf\" (UniqueName: \"kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.748654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748492 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.748654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.748654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.748654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.748865 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.748655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.849903 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.849864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.849921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.849955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6sf\" (UniqueName: \"kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850391 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850332 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850476 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.850540 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.850475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.852368 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.852346 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.852668 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.852648 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.859801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.859777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6sf\" (UniqueName: \"kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf\") pod \"e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:13.992719 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:13.992580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:14.133429 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:14.133395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:20:14.136529 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:20:14.136503 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod580150a0_b329_4112_80bd_f4a95bdf2f99.slice/crio-10c428f0d9e577a8360f48626c21f5ead460d9b7c3a81c0e635185a5902bfcd7 WatchSource:0}: Error finding container 10c428f0d9e577a8360f48626c21f5ead460d9b7c3a81c0e635185a5902bfcd7: Status 404 returned error can't find the container with id 10c428f0d9e577a8360f48626c21f5ead460d9b7c3a81c0e635185a5902bfcd7 Jul 17 13:20:14.607044 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:14.607009 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" event={"ID":"580150a0-b329-4112-80bd-f4a95bdf2f99","Type":"ContainerStarted","Data":"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28"} Jul 17 13:20:14.607244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:14.607058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" event={"ID":"580150a0-b329-4112-80bd-f4a95bdf2f99","Type":"ContainerStarted","Data":"10c428f0d9e577a8360f48626c21f5ead460d9b7c3a81c0e635185a5902bfcd7"} Jul 17 13:20:14.626398 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:14.626343 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" podStartSLOduration=1.626326132 podStartE2EDuration="1.626326132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:20:14.623923249 +0000 UTC m=+3464.568782760" watchObservedRunningTime="2026-07-17 13:20:14.626326132 +0000 UTC m=+3464.571185636" Jul 17 13:20:23.992865 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:23.992816 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:23.993271 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:23.992887 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:24.005050 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:24.005024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:20:24.664111 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:20:24.664078 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:21:04.998202 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:04.998099 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:21:20.019282 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.019235 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" podUID="580150a0-b329-4112-80bd-f4a95bdf2f99" containerName="main" containerID="cri-o://5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28" gracePeriod=45 Jul 17 13:21:20.279010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.278949 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:21:20.385159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385114 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385162 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6sf\" (UniqueName: \"kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385206 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385227 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385324 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home\") pod \"580150a0-b329-4112-80bd-f4a95bdf2f99\" (UID: \"580150a0-b329-4112-80bd-f4a95bdf2f99\") " Jul 17 13:21:20.385608 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385574 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:20.385770 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385739 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache" (OuterVolumeSpecName: "model-cache") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:20.385770 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.385753 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home" (OuterVolumeSpecName: "home") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:20.387560 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.387533 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm" (OuterVolumeSpecName: "dshm") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:20.387663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.387611 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:21:20.387663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.387614 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf" (OuterVolumeSpecName: "kube-api-access-gm6sf") pod "580150a0-b329-4112-80bd-f4a95bdf2f99" (UID: "580150a0-b329-4112-80bd-f4a95bdf2f99"). InnerVolumeSpecName "kube-api-access-gm6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:21:20.486054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486014 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.486054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486047 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.486054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486059 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.486307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486073 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6sf\" (UniqueName: \"kubernetes.io/projected/580150a0-b329-4112-80bd-f4a95bdf2f99-kube-api-access-gm6sf\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.486307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486085 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580150a0-b329-4112-80bd-f4a95bdf2f99-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.486307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.486121 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580150a0-b329-4112-80bd-f4a95bdf2f99-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:20.903745 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.903710 2568 generic.go:356] "Generic (PLEG): container finished" podID="580150a0-b329-4112-80bd-f4a95bdf2f99" containerID="5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28" exitCode=0 Jul 17 13:21:20.903745 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.903748 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" Jul 17 13:21:20.904013 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.903797 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" event={"ID":"580150a0-b329-4112-80bd-f4a95bdf2f99","Type":"ContainerDied","Data":"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28"} Jul 17 13:21:20.904013 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.903846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk" event={"ID":"580150a0-b329-4112-80bd-f4a95bdf2f99","Type":"ContainerDied","Data":"10c428f0d9e577a8360f48626c21f5ead460d9b7c3a81c0e635185a5902bfcd7"} Jul 17 13:21:20.904013 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.903869 2568 scope.go:117] "RemoveContainer" containerID="5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28" Jul 17 13:21:20.912149 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.912128 2568 scope.go:117] "RemoveContainer" containerID="5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28" Jul 17 13:21:20.912435 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:21:20.912417 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28\": container with ID starting with 5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28 not found: ID does not exist" containerID="5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28" Jul 17 13:21:20.912495 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.912448 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28"} err="failed to get container status \"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28\": rpc error: code = NotFound desc = could not find container \"5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28\": container with ID starting with 5eb4dbc33a272de5975b58f5089987d358d6f3b8f6022c112fe16f5fc1039a28 not found: ID does not exist" Jul 17 13:21:20.922803 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.922777 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:21:20.931169 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:20.931137 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6ad1d9a1/e2e-lifecycle-model-6ad1d9a1-kserve-6d7bfc8d9f-t5lfk"] Jul 17 13:21:22.586700 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:22.586658 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580150a0-b329-4112-80bd-f4a95bdf2f99" path="/var/lib/kubelet/pods/580150a0-b329-4112-80bd-f4a95bdf2f99/volumes" Jul 17 13:21:52.258660 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:52.258616 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:21:52.259290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:52.258900 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" containerID="cri-o://fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad" gracePeriod=30 Jul 17 13:21:57.518728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.516662 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530272 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530367 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530405 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf9sw\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530480 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530505 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530546 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530586 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.531313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.530612 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy\") pod \"71b929ff-2486-4e63-8490-ba67d4328b1c\" (UID: \"71b929ff-2486-4e63-8490-ba67d4328b1c\") " Jul 17 13:21:57.533794 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.532907 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:57.533794 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.533086 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:57.533794 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.533049 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data" (OuterVolumeSpecName: "istio-data") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:57.533794 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.533224 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:57.533794 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.533736 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:21:57.536077 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.536047 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:21:57.556041 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.549820 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw" (OuterVolumeSpecName: "kube-api-access-rf9sw") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "kube-api-access-rf9sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:21:57.556041 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.549898 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:21:57.560417 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.560351 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71b929ff-2486-4e63-8490-ba67d4328b1c" (UID: "71b929ff-2486-4e63-8490-ba67d4328b1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:21:57.632399 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632351 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632399 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632392 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632399 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632408 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632422 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b929ff-2486-4e63-8490-ba67d4328b1c-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632436 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632449 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632463 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632478 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b929ff-2486-4e63-8490-ba67d4328b1c-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:57.632718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:57.632512 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9sw\" (UniqueName: \"kubernetes.io/projected/71b929ff-2486-4e63-8490-ba67d4328b1c-kube-api-access-rf9sw\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:21:58.074859 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.074818 2568 generic.go:356] "Generic (PLEG): container finished" podID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerID="fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad" exitCode=0 Jul 17 13:21:58.074859 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.074873 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" Jul 17 13:21:58.075094 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.074886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerDied","Data":"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad"} Jul 17 13:21:58.075094 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.074941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2" event={"ID":"71b929ff-2486-4e63-8490-ba67d4328b1c","Type":"ContainerDied","Data":"7bbbf61dc1a16123ac03b353cc4e3ab9fd50496f49cc9ff96c4da5f6c7d88f98"} Jul 17 13:21:58.075094 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.074972 2568 scope.go:117] "RemoveContainer" containerID="fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad" Jul 17 13:21:58.085791 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.085767 2568 scope.go:117] "RemoveContainer" containerID="fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc" Jul 17 13:21:58.094506 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.094483 2568 scope.go:117] "RemoveContainer" containerID="fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad" Jul 17 13:21:58.094870 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:21:58.094851 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad\": container with ID starting with fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad not found: ID does not exist" containerID="fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad" Jul 17 13:21:58.094933 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.094881 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad"} err="failed to get container status \"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad\": rpc error: code = NotFound desc = could not find container \"fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad\": container with ID starting with fc7c33073adbb35078fa512ded0041282965e1bcd95be8321ffb68cc1dbf51ad not found: ID does not exist" Jul 17 13:21:58.094933 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.094903 2568 scope.go:117] "RemoveContainer" containerID="fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc" Jul 17 13:21:58.095189 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:21:58.095172 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc\": container with ID starting with fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc not found: ID does not exist" containerID="fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc" Jul 17 13:21:58.095250 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.095191 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc"} err="failed to get container status \"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc\": rpc error: code = NotFound desc = could not find container \"fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc\": container with ID starting with fda1988d381f652e7a31a629acdac5a941d3a0f4b48edf259623b78669db9dcc not found: ID does not exist" Jul 17 13:21:58.099337 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.099308 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:21:58.103354 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.103327 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ad1d9a1-openshift-default-5f74d48779-wpzr2"] Jul 17 13:21:58.585874 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:21:58.585844 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" path="/var/lib/kubelet/pods/71b929ff-2486-4e63-8490-ba67d4328b1c/volumes" Jul 17 13:22:49.276737 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.276702 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277048 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277058 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277067 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277072 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277082 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580150a0-b329-4112-80bd-f4a95bdf2f99" containerName="main" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277089 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="580150a0-b329-4112-80bd-f4a95bdf2f99" containerName="main" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277147 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277153 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b929ff-2486-4e63-8490-ba67d4328b1c" containerName="istio-proxy" Jul 17 13:22:49.277198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.277161 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="580150a0-b329-4112-80bd-f4a95bdf2f99" containerName="main" Jul 17 13:22:49.280131 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.280111 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.284348 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.284326 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae44b3-openshift-default-dockercfg-z2rv8\"" Jul 17 13:22:49.308901 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.308862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.308901 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.308909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.308939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvnw\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309015 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309044 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.309454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.309191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.316922 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.316887 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:22:49.410465 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvnw\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.410649 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.411212 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.410938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.411212 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.411005 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.411212 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.411089 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.411212 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.411108 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.411468 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.411450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.413313 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.413284 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.413406 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.413388 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.431793 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.431757 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.431927 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.431856 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvnw\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw\") pod \"e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.590810 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.590720 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:49.764056 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.764029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:22:49.765207 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:22:49.765180 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52209eb8_bcab_4343_a824_b102b3eddd02.slice/crio-e1a69155b7d2ea150c77b73dee3cc337a62051ef033f6406f88fe082fa6b112f WatchSource:0}: Error finding container e1a69155b7d2ea150c77b73dee3cc337a62051ef033f6406f88fe082fa6b112f: Status 404 returned error can't find the container with id e1a69155b7d2ea150c77b73dee3cc337a62051ef033f6406f88fe082fa6b112f Jul 17 13:22:49.767235 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.767202 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:49.767310 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.767268 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:49.767310 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:49.767296 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:50.305644 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:50.305611 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerStarted","Data":"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1"} Jul 17 13:22:50.306056 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:50.305663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerStarted","Data":"e1a69155b7d2ea150c77b73dee3cc337a62051ef033f6406f88fe082fa6b112f"} Jul 17 13:22:50.329055 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:50.328997 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podStartSLOduration=1.328979505 podStartE2EDuration="1.328979505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:50.327732172 +0000 UTC m=+3620.272591674" watchObservedRunningTime="2026-07-17 13:22:50.328979505 +0000 UTC m=+3620.273839009" Jul 17 13:22:50.591811 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:50.591727 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:50.597168 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:50.597139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:51.312100 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:51.312061 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:51.313191 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:51.313168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:22:57.663898 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.663859 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:22:57.666977 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.666958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.671321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.671297 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5dd9aa-openshift-default-dockercfg-nzp5c\"" Jul 17 13:22:57.788247 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788202 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788423 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788423 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r98\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788423 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788669 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.788669 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.788578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.818442 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.818410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:22:57.889957 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.889917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.889979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r98\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890399 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890408 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890428 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890720 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.890720 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.891010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.891010 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.890982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.892479 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.892455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.892735 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.892715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.904364 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.904335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.908703 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.908663 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r98\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98\") pod \"e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:57.977761 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:57.977644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:58.141663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.141637 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:22:58.144421 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:22:58.144396 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a552375_3934_41d3_a2b9_fcb3be2baf5d.slice/crio-d0c9c0b3ab4a810db344ebb2691945cbb070ea4e7795ab4c9da35f90b12f888d WatchSource:0}: Error finding container d0c9c0b3ab4a810db344ebb2691945cbb070ea4e7795ab4c9da35f90b12f888d: Status 404 returned error can't find the container with id d0c9c0b3ab4a810db344ebb2691945cbb070ea4e7795ab4c9da35f90b12f888d Jul 17 13:22:58.146393 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.146359 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:58.146480 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.146426 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:58.146480 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.146457 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:22:58.345941 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.345850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerStarted","Data":"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1"} Jul 17 13:22:58.345941 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.345908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerStarted","Data":"d0c9c0b3ab4a810db344ebb2691945cbb070ea4e7795ab4c9da35f90b12f888d"} Jul 17 13:22:58.387492 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.387429 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podStartSLOduration=1.387410717 podStartE2EDuration="1.387410717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:58.386725897 +0000 UTC m=+3628.331585402" watchObservedRunningTime="2026-07-17 13:22:58.387410717 +0000 UTC m=+3628.332270222" Jul 17 13:22:58.978037 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:58.977998 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:22:59.983823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:22:59.983795 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:00.355820 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:00.355723 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:00.356744 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:00.356727 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:07.354895 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:07.354855 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 13:23:07.355311 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:07.354919 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 13:23:08.397598 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.397553 2568 generic.go:356] "Generic (PLEG): container finished" podID="52209eb8-bcab-4343-a824-b102b3eddd02" containerID="bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1" exitCode=0 Jul 17 13:23:08.398144 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.397671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerDied","Data":"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1"} Jul 17 13:23:08.398216 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.398201 2568 scope.go:117] "RemoveContainer" containerID="bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1" Jul 17 13:23:08.400290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.399784 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:08.400290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.399874 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:08.400290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:08.399920 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:09.407430 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:09.407388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerStarted","Data":"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d"} Jul 17 13:23:09.591779 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:09.591740 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:23:09.592113 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:09.592081 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 17 13:23:09.592333 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:09.592149 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 17 13:23:10.592080 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:10.592041 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 17 13:23:10.592665 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:10.592631 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 17 13:23:11.591337 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:11.591294 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 17 13:23:11.591677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:11.591647 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 17 13:23:12.603708 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:12.599216 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 17 13:23:12.603708 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:12.599291 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 17 13:23:13.592248 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.592198 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 17 13:23:13.592460 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.592270 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 17 13:23:13.982046 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.982017 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:23:13.985198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.985179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:13.988662 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.988629 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae44b3\"/\"default-dockercfg-kpnln\"" Jul 17 13:23:13.988662 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.988641 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae44b3\"/\"test-model-ae44b3-kserve-self-signed-certs\"" Jul 17 13:23:13.988875 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.988660 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae44b3\"/\"kube-root-ca.crt\"" Jul 17 13:23:13.988875 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.988641 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae44b3\"/\"openshift-service-ca.crt\"" Jul 17 13:23:13.995289 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:13.995265 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:23:14.160085 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.160085 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.160315 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh26f\" (UniqueName: \"kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.160315 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.160315 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.160315 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.160278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261291 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261198 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261291 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261526 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261526 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261526 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh26f\" (UniqueName: \"kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261526 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.261978 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.261949 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.262180 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.262082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.262321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.262297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.265987 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.265929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.265987 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.265937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.272334 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.272302 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh26f\" (UniqueName: \"kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f\") pod \"test-model-ae44b3-kserve-5747b88859-th6f9\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.294821 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.294779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:14.447221 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.447056 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:23:14.449842 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:23:14.449808 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e31d15_5a01_460e_ad8b_e0df56cf3440.slice/crio-1bab80b587cc7c45aba2bfd9a599a0ecfce79a514ce8f61996f46913c52e5d57 WatchSource:0}: Error finding container 1bab80b587cc7c45aba2bfd9a599a0ecfce79a514ce8f61996f46913c52e5d57: Status 404 returned error can't find the container with id 1bab80b587cc7c45aba2bfd9a599a0ecfce79a514ce8f61996f46913c52e5d57 Jul 17 13:23:14.595473 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.595391 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:23:14.595825 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.595802 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:23:14.596544 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:14.596524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:23:15.357617 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.357575 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 17 13:23:15.358046 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.357634 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 17 13:23:15.450608 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.450577 2568 generic.go:356] "Generic (PLEG): container finished" podID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerID="cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1" exitCode=0 Jul 17 13:23:15.450777 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.450649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerDied","Data":"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1"} Jul 17 13:23:15.451142 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.451117 2568 scope.go:117] "RemoveContainer" containerID="cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1" Jul 17 13:23:15.452232 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.452213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" event={"ID":"d9e31d15-5a01-460e-ad8b-e0df56cf3440","Type":"ContainerStarted","Data":"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4"} Jul 17 13:23:15.452312 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.452244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" event={"ID":"d9e31d15-5a01-460e-ad8b-e0df56cf3440","Type":"ContainerStarted","Data":"1bab80b587cc7c45aba2bfd9a599a0ecfce79a514ce8f61996f46913c52e5d57"} Jul 17 13:23:15.452312 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.452233 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:15.452381 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.452319 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:15.452381 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.452364 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:23:15.491665 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:15.491601 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" podStartSLOduration=2.491579523 podStartE2EDuration="2.491579523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:23:15.490901472 +0000 UTC m=+3645.435760977" watchObservedRunningTime="2026-07-17 13:23:15.491579523 +0000 UTC m=+3645.436439029" Jul 17 13:23:16.458280 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:16.458238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerStarted","Data":"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66"} Jul 17 13:23:16.978556 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:16.978519 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:16.978769 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:16.978752 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 17 13:23:16.978828 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:16.978798 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 17 13:23:17.978631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:17.978593 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 17 13:23:17.979048 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:17.978649 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 17 13:23:18.979024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:18.978984 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 17 13:23:18.979423 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:18.979053 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 17 13:23:19.978861 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:19.978824 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 17 13:23:19.979054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:19.978882 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 17 13:23:20.982813 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:20.982779 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:20.983274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:20.983128 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:20.984008 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:20.983992 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:23:24.295494 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:24.295455 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:24.296036 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:24.295511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:24.308154 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:24.308125 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:23:24.506363 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:23:24.506326 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:24:15.839524 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.839487 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:15.841954 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.841938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.847513 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.847479 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dd9aa\"/\"test-model-5dd9aa-kserve-self-signed-certs\"" Jul 17 13:24:15.847719 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.847679 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dd9aa\"/\"kube-root-ca.crt\"" Jul 17 13:24:15.854105 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.854081 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dd9aa\"/\"default-dockercfg-bk8sz\"" Jul 17 13:24:15.854207 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.854184 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dd9aa\"/\"openshift-service-ca.crt\"" Jul 17 13:24:15.869190 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.869149 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:15.959348 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.959348 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.959577 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.959577 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzw2\" (UniqueName: \"kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.959577 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:15.959577 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:15.959548 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060373 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060373 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060652 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060652 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzw2\" (UniqueName: \"kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060652 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060652 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.060963 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060939 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.061007 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.060948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.061045 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.061006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.062760 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.062738 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.063105 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.063086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.069476 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.069452 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzw2\" (UniqueName: \"kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2\") pod \"test-model-5dd9aa-kserve-58b487fd85-xt9cp\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.151351 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.151311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:16.285954 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.285928 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:16.288379 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:24:16.288339 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4374d7f6_009a_4f7c_a3bf_70d5b3c7d504.slice/crio-7aaada12b1e9abd435761bdc9d837e8c87690060a5a60e224ba762ff705cd392 WatchSource:0}: Error finding container 7aaada12b1e9abd435761bdc9d837e8c87690060a5a60e224ba762ff705cd392: Status 404 returned error can't find the container with id 7aaada12b1e9abd435761bdc9d837e8c87690060a5a60e224ba762ff705cd392 Jul 17 13:24:16.729753 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.729718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" event={"ID":"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504","Type":"ContainerStarted","Data":"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847"} Jul 17 13:24:16.729922 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.729762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" event={"ID":"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504","Type":"ContainerStarted","Data":"7aaada12b1e9abd435761bdc9d837e8c87690060a5a60e224ba762ff705cd392"} Jul 17 13:24:16.753624 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:16.753572 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" podStartSLOduration=1.753555875 podStartE2EDuration="1.753555875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:24:16.750591015 +0000 UTC m=+3706.695450518" watchObservedRunningTime="2026-07-17 13:24:16.753555875 +0000 UTC m=+3706.698415380" Jul 17 13:24:26.152402 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:26.152367 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:26.152795 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:26.152423 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:26.165160 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:26.165132 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:26.793945 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:26.793906 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:39.539300 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:39.539261 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:24:40.486275 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:40.486212 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:54.559621 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.559577 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" podUID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" containerName="main" containerID="cri-o://256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4" gracePeriod=45 Jul 17 13:24:54.816072 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.816013 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:24:54.914186 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.914153 2568 generic.go:356] "Generic (PLEG): container finished" podID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" containerID="256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4" exitCode=0 Jul 17 13:24:54.914186 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.914192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" event={"ID":"d9e31d15-5a01-460e-ad8b-e0df56cf3440","Type":"ContainerDied","Data":"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4"} Jul 17 13:24:54.914419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.914198 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" Jul 17 13:24:54.914419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.914219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9" event={"ID":"d9e31d15-5a01-460e-ad8b-e0df56cf3440","Type":"ContainerDied","Data":"1bab80b587cc7c45aba2bfd9a599a0ecfce79a514ce8f61996f46913c52e5d57"} Jul 17 13:24:54.914419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.914236 2568 scope.go:117] "RemoveContainer" containerID="256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4" Jul 17 13:24:54.922596 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.922569 2568 scope.go:117] "RemoveContainer" containerID="256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4" Jul 17 13:24:54.922988 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:24:54.922969 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4\": container with ID starting with 256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4 not found: ID does not exist" containerID="256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4" Jul 17 13:24:54.923040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.922997 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4"} err="failed to get container status \"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4\": rpc error: code = NotFound desc = could not find container \"256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4\": container with ID starting with 256808c25527f588b7d9da7d68db7c1c0d0423f2ccf5263f5d18a5b016bed2c4 not found: ID does not exist" Jul 17 13:24:54.939344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939311 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939463 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939373 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh26f\" (UniqueName: \"kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939463 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939397 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939463 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939579 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939523 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939625 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939592 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir\") pod \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\" (UID: \"d9e31d15-5a01-460e-ad8b-e0df56cf3440\") " Jul 17 13:24:54.939678 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939618 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:54.939908 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939864 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home" (OuterVolumeSpecName: "home") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:54.939908 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939896 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:54.940096 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.939908 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:54.941922 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.941899 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:24:54.942011 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.941915 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm" (OuterVolumeSpecName: "dshm") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:54.942011 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:54.941932 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f" (OuterVolumeSpecName: "kube-api-access-rh26f") pod "d9e31d15-5a01-460e-ad8b-e0df56cf3440" (UID: "d9e31d15-5a01-460e-ad8b-e0df56cf3440"). InnerVolumeSpecName "kube-api-access-rh26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:24:55.040844 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.040809 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.040844 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.040840 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.040844 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.040852 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh26f\" (UniqueName: \"kubernetes.io/projected/d9e31d15-5a01-460e-ad8b-e0df56cf3440-kube-api-access-rh26f\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.041087 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.040861 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e31d15-5a01-460e-ad8b-e0df56cf3440-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.041087 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.040870 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e31d15-5a01-460e-ad8b-e0df56cf3440-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.236105 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.236071 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:24:55.240572 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.240537 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae44b3/test-model-ae44b3-kserve-5747b88859-th6f9"] Jul 17 13:24:55.506761 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.506649 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" podUID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" containerName="main" containerID="cri-o://fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847" gracePeriod=45 Jul 17 13:24:55.755889 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.755864 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:55.852962 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.852868 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.852962 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.852923 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.852962 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.852961 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.853257 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.852983 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.853257 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853090 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.853257 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853143 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqzw2\" (UniqueName: \"kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2\") pod \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\" (UID: \"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504\") " Jul 17 13:24:55.853257 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853154 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache" (OuterVolumeSpecName: "model-cache") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:55.853428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853287 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home" (OuterVolumeSpecName: "home") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:55.853428 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853325 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:55.853520 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853454 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.853520 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853467 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.853520 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.853494 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.855383 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.855361 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm" (OuterVolumeSpecName: "dshm") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:24:55.855489 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.855405 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2" (OuterVolumeSpecName: "kube-api-access-rqzw2") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "kube-api-access-rqzw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:24:55.855489 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.855456 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" (UID: "4374d7f6-009a-4f7c-a3bf-70d5b3c7d504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:24:55.921889 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.921853 2568 generic.go:356] "Generic (PLEG): container finished" podID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" containerID="fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847" exitCode=0 Jul 17 13:24:55.921889 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.921893 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" Jul 17 13:24:55.922131 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.921895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" event={"ID":"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504","Type":"ContainerDied","Data":"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847"} Jul 17 13:24:55.922131 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.922033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp" event={"ID":"4374d7f6-009a-4f7c-a3bf-70d5b3c7d504","Type":"ContainerDied","Data":"7aaada12b1e9abd435761bdc9d837e8c87690060a5a60e224ba762ff705cd392"} Jul 17 13:24:55.922131 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.922065 2568 scope.go:117] "RemoveContainer" containerID="fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847" Jul 17 13:24:55.930807 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.930786 2568 scope.go:117] "RemoveContainer" containerID="fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847" Jul 17 13:24:55.931064 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:24:55.931044 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847\": container with ID starting with fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847 not found: ID does not exist" containerID="fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847" Jul 17 13:24:55.931122 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.931073 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847"} err="failed to get container status \"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847\": rpc error: code = NotFound desc = could not find container \"fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847\": container with ID starting with fb2098870602b7f82c713415e19ca0da5150679b58c6718fe5724db0c1036847 not found: ID does not exist" Jul 17 13:24:55.942836 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.942804 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:55.946514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.946492 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5dd9aa/test-model-5dd9aa-kserve-58b487fd85-xt9cp"] Jul 17 13:24:55.954356 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.954323 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.954356 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.954359 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzw2\" (UniqueName: \"kubernetes.io/projected/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-kube-api-access-rqzw2\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:55.954481 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:55.954369 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:24:56.586999 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:56.586967 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" path="/var/lib/kubelet/pods/4374d7f6-009a-4f7c-a3bf-70d5b3c7d504/volumes" Jul 17 13:24:56.587361 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:24:56.587348 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" path="/var/lib/kubelet/pods/d9e31d15-5a01-460e-ad8b-e0df56cf3440/volumes" Jul 17 13:25:06.444855 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:06.444816 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:25:06.445306 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:06.445029 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" containerID="cri-o://0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d" gracePeriod=30 Jul 17 13:25:11.706426 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.706359 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:25:11.803401 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803364 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803401 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803403 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803483 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803631 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803634 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803677 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:11.803827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803714 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.803827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsvnw\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.804029 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803835 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:11.804029 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803854 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token\") pod \"52209eb8-bcab-4343-a824-b102b3eddd02\" (UID: \"52209eb8-bcab-4343-a824-b102b3eddd02\") " Jul 17 13:25:11.804029 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.803935 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:11.804029 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804022 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:25:11.804255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804192 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.804255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804210 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.804255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804224 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.804255 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804236 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52209eb8-bcab-4343-a824-b102b3eddd02-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.804784 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.804757 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data" (OuterVolumeSpecName: "istio-data") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:11.805993 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.805962 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:11.805993 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.805963 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:25:11.806153 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.806077 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token" (OuterVolumeSpecName: "istio-token") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:25:11.806212 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.806170 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw" (OuterVolumeSpecName: "kube-api-access-zsvnw") pod "52209eb8-bcab-4343-a824-b102b3eddd02" (UID: "52209eb8-bcab-4343-a824-b102b3eddd02"). InnerVolumeSpecName "kube-api-access-zsvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:25:11.905249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.905209 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.905249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.905241 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52209eb8-bcab-4343-a824-b102b3eddd02-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.905249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.905250 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52209eb8-bcab-4343-a824-b102b3eddd02-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.905249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.905258 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsvnw\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-kube-api-access-zsvnw\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:11.905505 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:11.905268 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52209eb8-bcab-4343-a824-b102b3eddd02-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:12.002275 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.002191 2568 generic.go:356] "Generic (PLEG): container finished" podID="52209eb8-bcab-4343-a824-b102b3eddd02" containerID="0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d" exitCode=0 Jul 17 13:25:12.002275 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.002237 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" Jul 17 13:25:12.002275 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.002260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerDied","Data":"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d"} Jul 17 13:25:12.002585 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.002303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2" event={"ID":"52209eb8-bcab-4343-a824-b102b3eddd02","Type":"ContainerDied","Data":"e1a69155b7d2ea150c77b73dee3cc337a62051ef033f6406f88fe082fa6b112f"} Jul 17 13:25:12.002585 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.002323 2568 scope.go:117] "RemoveContainer" containerID="0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d" Jul 17 13:25:12.011876 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.011854 2568 scope.go:117] "RemoveContainer" containerID="bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1" Jul 17 13:25:12.019602 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.019583 2568 scope.go:117] "RemoveContainer" containerID="0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d" Jul 17 13:25:12.019894 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:25:12.019876 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d\": container with ID starting with 0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d not found: ID does not exist" containerID="0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d" Jul 17 13:25:12.019935 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.019903 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d"} err="failed to get container status \"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d\": rpc error: code = NotFound desc = could not find container \"0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d\": container with ID starting with 0432498bdaed7971ea167d469359a79e985568e7e12556afdd91ddf231d42d6d not found: ID does not exist" Jul 17 13:25:12.019935 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.019922 2568 scope.go:117] "RemoveContainer" containerID="bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1" Jul 17 13:25:12.020166 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:25:12.020146 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1\": container with ID starting with bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1 not found: ID does not exist" containerID="bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1" Jul 17 13:25:12.020223 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.020173 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1"} err="failed to get container status \"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1\": rpc error: code = NotFound desc = could not find container \"bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1\": container with ID starting with bc38497f7976f8df37c6be31a4ee34f71e47ba8d771ba54a26043d55adca2fe1 not found: ID does not exist" Jul 17 13:25:12.025458 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.025434 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:25:12.032285 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.032262 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae44b3-openshift-default-855484b6c9-k2lh2"] Jul 17 13:25:12.592216 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:12.592177 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" path="/var/lib/kubelet/pods/52209eb8-bcab-4343-a824-b102b3eddd02/volumes" Jul 17 13:25:32.635309 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:32.635271 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:25:32.635837 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:32.635490 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" containerID="cri-o://1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66" gracePeriod=30 Jul 17 13:25:33.609000 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.608962 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:25:33.609370 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609353 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" containerName="main" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609373 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" containerName="main" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609383 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" containerName="main" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609391 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" containerName="main" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609408 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609417 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609430 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609438 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609854 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609525 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609854 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609538 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52209eb8-bcab-4343-a824-b102b3eddd02" containerName="istio-proxy" Jul 17 13:25:33.609854 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609551 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e31d15-5a01-460e-ad8b-e0df56cf3440" containerName="main" Jul 17 13:25:33.609854 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.609561 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4374d7f6-009a-4f7c-a3bf-70d5b3c7d504" containerName="main" Jul 17 13:25:33.614004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.613981 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.626984 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.626962 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-771703-openshift-default-dockercfg-f9glv\"" Jul 17 13:25:33.654290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.654258 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:25:33.696783 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.696783 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697003 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmdl\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697003 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697003 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697003 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697003 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.696976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697170 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.697030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.697170 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.697050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.798595 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.798595 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798595 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.798886 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmdl\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.798886 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798773 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.798886 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798990 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.798996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799273 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.799061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799273 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.799266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799506 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.799390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799506 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.799450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.799660 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.799549 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.801420 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.801400 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.801544 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.801526 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.816501 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.816467 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmdl\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.817557 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.817533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token\") pod \"e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:33.923367 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:33.923324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:34.074141 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:34.074119 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:25:34.075621 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:25:34.075594 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfea0f7_b4ed_4bfb_aca6_c27824a9ae59.slice/crio-48d39674de8d8e8e189fff758812208b13e0fad9dbf65850cb3ec81d630928ed WatchSource:0}: Error finding container 48d39674de8d8e8e189fff758812208b13e0fad9dbf65850cb3ec81d630928ed: Status 404 returned error can't find the container with id 48d39674de8d8e8e189fff758812208b13e0fad9dbf65850cb3ec81d630928ed Jul 17 13:25:34.077632 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:34.077603 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:34.077720 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:34.077664 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:34.077720 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:34.077710 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:34.101344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:34.101314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerStarted","Data":"48d39674de8d8e8e189fff758812208b13e0fad9dbf65850cb3ec81d630928ed"} Jul 17 13:25:35.106880 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.106833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerStarted","Data":"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538"} Jul 17 13:25:35.135455 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.135403 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podStartSLOduration=2.135383455 podStartE2EDuration="2.135383455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:35.134445235 +0000 UTC m=+3785.079304740" watchObservedRunningTime="2026-07-17 13:25:35.135383455 +0000 UTC m=+3785.080242961" Jul 17 13:25:35.923838 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.923798 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:35.928974 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.928950 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:35.986145 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.986102 2568 patch_prober.go:28] interesting pod/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 13:25:35.986311 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:35.986162 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 13:25:36.111472 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:36.111441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:36.112416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:36.112398 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:37.889434 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.889408 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:25:37.938560 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938477 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938560 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938520 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8r98\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938595 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938627 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938710 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938744 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.938815 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938810 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.939094 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938834 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token\") pod \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\" (UID: \"8a552375-3934-41d3-a2b9-fcb3be2baf5d\") " Jul 17 13:25:37.939094 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.938892 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:37.939209 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939111 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:37.939209 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939133 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:37.939209 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939149 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:25:37.939370 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939213 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:37.939370 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939232 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:37.939464 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.939380 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:37.941062 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.941033 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98" (OuterVolumeSpecName: "kube-api-access-p8r98") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "kube-api-access-p8r98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:25:37.941271 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.941253 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:25:37.941455 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.941440 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:25:37.941796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:37.941777 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a552375-3934-41d3-a2b9-fcb3be2baf5d" (UID: "8a552375-3934-41d3-a2b9-fcb3be2baf5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:25:38.040120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040082 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040112 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040122 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040130 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040445 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040137 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040445 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040145 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a552375-3934-41d3-a2b9-fcb3be2baf5d-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.040445 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.040152 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8r98\" (UniqueName: \"kubernetes.io/projected/8a552375-3934-41d3-a2b9-fcb3be2baf5d-kube-api-access-p8r98\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:25:38.121145 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.121106 2568 generic.go:356] "Generic (PLEG): container finished" podID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerID="1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66" exitCode=0 Jul 17 13:25:38.121338 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.121154 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" Jul 17 13:25:38.121338 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.121194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerDied","Data":"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66"} Jul 17 13:25:38.121338 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.121278 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl" event={"ID":"8a552375-3934-41d3-a2b9-fcb3be2baf5d","Type":"ContainerDied","Data":"d0c9c0b3ab4a810db344ebb2691945cbb070ea4e7795ab4c9da35f90b12f888d"} Jul 17 13:25:38.121338 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.121305 2568 scope.go:117] "RemoveContainer" containerID="1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66" Jul 17 13:25:38.130944 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.130922 2568 scope.go:117] "RemoveContainer" containerID="cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1" Jul 17 13:25:38.139139 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.139119 2568 scope.go:117] "RemoveContainer" containerID="1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66" Jul 17 13:25:38.139401 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:25:38.139368 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66\": container with ID starting with 1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66 not found: ID does not exist" containerID="1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66" Jul 17 13:25:38.139493 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.139405 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66"} err="failed to get container status \"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66\": rpc error: code = NotFound desc = could not find container \"1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66\": container with ID starting with 1dc0808ba55b945ba2eae78eed21858896a51628dc4f2f9801411879b74e7e66 not found: ID does not exist" Jul 17 13:25:38.139493 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.139421 2568 scope.go:117] "RemoveContainer" containerID="cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1" Jul 17 13:25:38.139611 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:25:38.139596 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1\": container with ID starting with cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1 not found: ID does not exist" containerID="cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1" Jul 17 13:25:38.139654 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.139613 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1"} err="failed to get container status \"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1\": rpc error: code = NotFound desc = could not find container \"cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1\": container with ID starting with cda1da0022bfb79077936a1db1e3c2285272b8162cfc66632e5672e56160fea1 not found: ID does not exist" Jul 17 13:25:38.146079 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.146055 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:25:38.148742 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.148721 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd9aa-openshift-default-5d9f7b5d67-76wpl"] Jul 17 13:25:38.588801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:38.588769 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" path="/var/lib/kubelet/pods/8a552375-3934-41d3-a2b9-fcb3be2baf5d/volumes" Jul 17 13:25:42.359397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359357 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:25:42.359789 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359771 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:25:42.359789 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359789 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:25:42.359879 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359809 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:25:42.359879 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359814 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:25:42.359879 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.359871 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:25:42.364213 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.364191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.366791 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.366769 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6fa40d-openshift-default-dockercfg-hxfpg\"" Jul 17 13:25:42.378886 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.378857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:25:42.485576 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485782 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485956 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485784 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485956 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gv2\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.485956 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.485876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.586801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gv2\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587108 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587108 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.586979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587108 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.587084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587308 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.587291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587373 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.587310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.587804 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.587783 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.589109 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.589090 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.589304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.589285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.594201 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.594173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.594297 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.594206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gv2\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2\") pod \"e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.674666 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.674630 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:42.815672 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.815542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:25:42.818067 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:25:42.818036 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06094b63_f0a1_4401_8fcb_96f79ca6ee0c.slice/crio-e10e4eac4999e4a89b241456e6cbd3c8c7e789df622cb07859248586a8929859 WatchSource:0}: Error finding container e10e4eac4999e4a89b241456e6cbd3c8c7e789df622cb07859248586a8929859: Status 404 returned error can't find the container with id e10e4eac4999e4a89b241456e6cbd3c8c7e789df622cb07859248586a8929859 Jul 17 13:25:42.820178 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.820147 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:42.820249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.820210 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:42.820249 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:42.820241 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:43.145283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:43.145252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerStarted","Data":"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32"} Jul 17 13:25:43.145465 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:43.145305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerStarted","Data":"e10e4eac4999e4a89b241456e6cbd3c8c7e789df622cb07859248586a8929859"} Jul 17 13:25:43.165053 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:43.165002 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podStartSLOduration=1.164987179 podStartE2EDuration="1.164987179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:43.163744757 +0000 UTC m=+3793.108604264" watchObservedRunningTime="2026-07-17 13:25:43.164987179 +0000 UTC m=+3793.109846682" Jul 17 13:25:43.675225 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:43.675184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:43.680186 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:43.680161 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:44.150303 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:44.150276 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:44.151226 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:44.151207 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:25:50.181199 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.181167 2568 generic.go:356] "Generic (PLEG): container finished" podID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerID="40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538" exitCode=0 Jul 17 13:25:50.181624 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.181250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerDied","Data":"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538"} Jul 17 13:25:50.181667 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.181639 2568 scope.go:117] "RemoveContainer" containerID="40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538" Jul 17 13:25:50.182643 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.182615 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:50.182734 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.182714 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:50.182775 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:50.182759 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:25:51.111803 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:51.111747 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:51.191220 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:51.191176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerStarted","Data":"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e"} Jul 17 13:25:51.924038 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:51.924003 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:51.925281 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:51.925254 2568 patch_prober.go:28] interesting pod/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 17 13:25:51.925396 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:51.925362 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 17 13:25:52.924336 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:52.924294 2568 patch_prober.go:28] interesting pod/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 17 13:25:52.924745 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:52.924353 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 17 13:25:53.924353 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:53.924310 2568 patch_prober.go:28] interesting pod/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 17 13:25:53.924757 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:53.924375 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 17 13:25:54.923907 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:54.923858 2568 patch_prober.go:28] interesting pod/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 17 13:25:54.924179 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:54.923932 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 17 13:25:55.928155 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:55.928118 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:55.928626 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:55.928607 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:25:55.929383 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:25:55.929362 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:26:00.191897 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:00.191855 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 13:26:00.192274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:00.191911 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 13:26:01.252137 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.252105 2568 generic.go:356] "Generic (PLEG): container finished" podID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerID="d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32" exitCode=0 Jul 17 13:26:01.252639 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.252185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerDied","Data":"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32"} Jul 17 13:26:01.252777 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.252760 2568 scope.go:117] "RemoveContainer" containerID="d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32" Jul 17 13:26:01.253826 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.253799 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:01.253891 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.253863 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:01.253928 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:01.253895 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:02.258656 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:02.258624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerStarted","Data":"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc"} Jul 17 13:26:02.675041 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:02.675006 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:02.675274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:02.675253 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 17 13:26:02.675349 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:02.675331 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 17 13:26:03.675790 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:03.675756 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 17 13:26:03.676173 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:03.675825 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 17 13:26:04.675389 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:04.675351 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 17 13:26:04.675572 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:04.675408 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 17 13:26:05.675967 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:05.675932 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 17 13:26:05.676380 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:05.675988 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 17 13:26:06.678725 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:06.678675 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:06.679165 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:06.678946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:06.679539 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:06.679524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:23.410997 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:23.410962 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:26:23.411443 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:23.411217 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" containerID="cri-o://8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e" gracePeriod=30 Jul 17 13:26:25.931320 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:25.931274 2568 patch_prober.go:28] interesting pod/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 13:26:25.931803 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:25.931348 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 13:26:28.660903 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.660879 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:26:28.758754 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758632 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.758754 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758677 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.758754 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758748 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758778 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758817 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758836 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758863 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvmdl\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758936 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759024 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.758964 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs\") pod \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\" (UID: \"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59\") " Jul 17 13:26:28.759307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759039 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:28.759307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759258 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:28.759307 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759284 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:26:28.759447 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759353 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.759447 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759369 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.759576 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.759556 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:28.760338 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.760310 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data" (OuterVolumeSpecName: "istio-data") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:28.761264 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.761242 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:28.761402 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.761373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl" (OuterVolumeSpecName: "kube-api-access-wvmdl") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "kube-api-access-wvmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:26:28.761491 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.761414 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:26:28.761491 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.761438 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token" (OuterVolumeSpecName: "istio-token") pod "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" (UID: "7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:26:28.860069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860013 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860062 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860074 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860087 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860098 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860107 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:28.860321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:28.860117 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmdl\" (UniqueName: \"kubernetes.io/projected/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59-kube-api-access-wvmdl\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:29.380965 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.380928 2568 generic.go:356] "Generic (PLEG): container finished" podID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerID="8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e" exitCode=0 Jul 17 13:26:29.381120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.380970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerDied","Data":"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e"} Jul 17 13:26:29.381120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.380995 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" Jul 17 13:26:29.381120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.381020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl" event={"ID":"7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59","Type":"ContainerDied","Data":"48d39674de8d8e8e189fff758812208b13e0fad9dbf65850cb3ec81d630928ed"} Jul 17 13:26:29.381120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.381043 2568 scope.go:117] "RemoveContainer" containerID="8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e" Jul 17 13:26:29.390115 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.390098 2568 scope.go:117] "RemoveContainer" containerID="40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538" Jul 17 13:26:29.398567 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.398250 2568 scope.go:117] "RemoveContainer" containerID="8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e" Jul 17 13:26:29.398970 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:26:29.398948 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e\": container with ID starting with 8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e not found: ID does not exist" containerID="8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e" Jul 17 13:26:29.399046 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.398978 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e"} err="failed to get container status \"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e\": rpc error: code = NotFound desc = could not find container \"8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e\": container with ID starting with 8436efa2bdbc30a131623b018e66eb09e44d03fac99c369f5a5537afb6617e1e not found: ID does not exist" Jul 17 13:26:29.399046 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.398999 2568 scope.go:117] "RemoveContainer" containerID="40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538" Jul 17 13:26:29.399427 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:26:29.399394 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538\": container with ID starting with 40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538 not found: ID does not exist" containerID="40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538" Jul 17 13:26:29.399532 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.399431 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538"} err="failed to get container status \"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538\": rpc error: code = NotFound desc = could not find container \"40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538\": container with ID starting with 40e2616bc709680ea2c4b887285630dad4f9aa64755669166854d7c41e796538 not found: ID does not exist" Jul 17 13:26:29.416747 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.416714 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:26:29.422783 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:29.422755 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-771703-openshift-default-8445dc9c5c-7wbcl"] Jul 17 13:26:30.587442 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:30.587414 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" path="/var/lib/kubelet/pods/7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59/volumes" Jul 17 13:26:49.643605 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:49.643570 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:26:49.644077 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:49.643871 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" containerID="cri-o://f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc" gracePeriod=30 Jul 17 13:26:50.605353 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605323 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:26:50.605675 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605655 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:50.605675 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605670 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:50.605841 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605762 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a552375-3934-41d3-a2b9-fcb3be2baf5d" containerName="istio-proxy" Jul 17 13:26:50.605841 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605774 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:50.605841 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.605779 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:50.608663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.608647 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.610950 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.610929 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e92eab-openshift-default-dockercfg-x5p9f\"" Jul 17 13:26:50.619742 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.619711 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:26:50.663619 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.663619 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663665 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.664066 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.663896 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765382 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765337 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765382 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.765640 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.766056 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.766056 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.765899 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.766218 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.766182 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.766322 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.766300 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.766454 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.766433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.768637 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.768615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.768788 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.768725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.777777 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.777746 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.778452 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.778428 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token\") pod \"e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:50.918924 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:50.918885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:51.063305 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.063280 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:26:51.065980 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:26:51.065957 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e30f56_95c3_467d_a134_9dad4883a12e.slice/crio-5dd75318fa0e2a58a2498fcde4be416adb3671b7e3b1611853d5369c7a501d3e WatchSource:0}: Error finding container 5dd75318fa0e2a58a2498fcde4be416adb3671b7e3b1611853d5369c7a501d3e: Status 404 returned error can't find the container with id 5dd75318fa0e2a58a2498fcde4be416adb3671b7e3b1611853d5369c7a501d3e Jul 17 13:26:51.068087 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.068053 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:51.068169 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.068121 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:51.068169 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.068150 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:51.483309 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.483214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerStarted","Data":"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96"} Jul 17 13:26:51.483478 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.483341 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerStarted","Data":"5dd75318fa0e2a58a2498fcde4be416adb3671b7e3b1611853d5369c7a501d3e"} Jul 17 13:26:51.506886 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.506834 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podStartSLOduration=1.506819592 podStartE2EDuration="1.506819592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:51.505268524 +0000 UTC m=+3861.450128032" watchObservedRunningTime="2026-07-17 13:26:51.506819592 +0000 UTC m=+3861.451679096" Jul 17 13:26:51.681257 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.681216 2568 patch_prober.go:28] interesting pod/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 13:26:51.681623 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.681285 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 13:26:51.919321 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:51.919291 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:52.924254 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:52.924224 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:53.492744 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:53.492711 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:53.493669 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:53.493648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:26:54.888562 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:54.888538 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:55.006278 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006185 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006278 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006258 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006287 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006311 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006356 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006380 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006432 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006509 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006468 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89gv2\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006510 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data\") pod \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\" (UID: \"06094b63-f0a1-4401-8fcb-96f79ca6ee0c\") " Jul 17 13:26:55.006827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006594 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:55.006827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006645 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:55.006827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006796 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:26:55.007031 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006883 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.007031 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006982 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:55.007031 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.006995 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.007031 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.007013 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.007616 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.007595 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:55.008743 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.008720 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2" (OuterVolumeSpecName: "kube-api-access-89gv2") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "kube-api-access-89gv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:26:55.008847 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.008768 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:26:55.008938 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.008922 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:26:55.008987 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.008949 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "06094b63-f0a1-4401-8fcb-96f79ca6ee0c" (UID: "06094b63-f0a1-4401-8fcb-96f79ca6ee0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:26:55.108304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108271 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.108304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108301 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gv2\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-kube-api-access-89gv2\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.108304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108310 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.108536 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108318 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.108536 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108326 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.108536 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.108333 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06094b63-f0a1-4401-8fcb-96f79ca6ee0c-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:26:55.503220 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.503187 2568 generic.go:356] "Generic (PLEG): container finished" podID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerID="f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc" exitCode=0 Jul 17 13:26:55.503397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.503236 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" Jul 17 13:26:55.503397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.503261 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerDied","Data":"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc"} Jul 17 13:26:55.503397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.503323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd" event={"ID":"06094b63-f0a1-4401-8fcb-96f79ca6ee0c","Type":"ContainerDied","Data":"e10e4eac4999e4a89b241456e6cbd3c8c7e789df622cb07859248586a8929859"} Jul 17 13:26:55.503397 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.503345 2568 scope.go:117] "RemoveContainer" containerID="f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc" Jul 17 13:26:55.511888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.511870 2568 scope.go:117] "RemoveContainer" containerID="d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32" Jul 17 13:26:55.519593 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.519577 2568 scope.go:117] "RemoveContainer" containerID="f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc" Jul 17 13:26:55.519867 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:26:55.519847 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc\": container with ID starting with f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc not found: ID does not exist" containerID="f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc" Jul 17 13:26:55.519932 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.519874 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc"} err="failed to get container status \"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc\": rpc error: code = NotFound desc = could not find container \"f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc\": container with ID starting with f8006c9e6aadaae90cb25f13efd18e15027db25a62280528148b8e552abaf0dc not found: ID does not exist" Jul 17 13:26:55.519932 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.519893 2568 scope.go:117] "RemoveContainer" containerID="d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32" Jul 17 13:26:55.520157 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:26:55.520138 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32\": container with ID starting with d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32 not found: ID does not exist" containerID="d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32" Jul 17 13:26:55.520205 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.520164 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32"} err="failed to get container status \"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32\": rpc error: code = NotFound desc = could not find container \"d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32\": container with ID starting with d068b0a9d8b0ca910c072a33834e86b38e84c359fe73b486557a2b2222968f32 not found: ID does not exist" Jul 17 13:26:55.524760 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.524740 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:26:55.530118 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:55.530098 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6fa40d-openshift-default-86d9958467-pnhwd"] Jul 17 13:26:56.586063 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:56.586022 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" path="/var/lib/kubelet/pods/06094b63-f0a1-4401-8fcb-96f79ca6ee0c/volumes" Jul 17 13:26:58.673937 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.673904 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:26:58.674294 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674242 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:26:58.674294 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674253 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:26:58.674294 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674265 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:26:58.674294 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674274 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:26:58.674419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674303 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:58.674419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674312 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfea0f7-b4ed-4bfb-aca6-c27824a9ae59" containerName="istio-proxy" Jul 17 13:26:58.674419 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.674395 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:26:58.679050 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.679031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.681462 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.681439 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5c3765-openshift-default-dockercfg-p6dsl\"" Jul 17 13:26:58.692616 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.692593 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:26:58.841894 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.841843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc7z\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842175 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842287 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842230 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842369 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842445 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842502 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842562 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842501 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842562 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.842644 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.842568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.943756 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.943756 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.943756 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc7z\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944030 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.943929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944296 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.944113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944332 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.944319 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944480 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.944444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944480 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.944458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.944573 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.944507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.946744 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.946717 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.946889 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.946871 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.951938 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.951916 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc7z\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.952037 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.952017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token\") pod \"e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:58.989136 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:58.989104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:26:59.150180 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.150144 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:26:59.153252 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:26:59.153224 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab91a10_eaac_44a5_9a31_a1e0c47de0f3.slice/crio-aa04c77f9a6ff3bc9a5ee6d54f0251fe91836fa5684641af12647c04c581d89b WatchSource:0}: Error finding container aa04c77f9a6ff3bc9a5ee6d54f0251fe91836fa5684641af12647c04c581d89b: Status 404 returned error can't find the container with id aa04c77f9a6ff3bc9a5ee6d54f0251fe91836fa5684641af12647c04c581d89b Jul 17 13:26:59.155705 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.155648 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:59.155798 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.155731 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:59.155798 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.155763 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:26:59.522052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.521958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerStarted","Data":"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a"} Jul 17 13:26:59.522052 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.522014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerStarted","Data":"aa04c77f9a6ff3bc9a5ee6d54f0251fe91836fa5684641af12647c04c581d89b"} Jul 17 13:26:59.544989 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.544932 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podStartSLOduration=1.5449156400000001 podStartE2EDuration="1.54491564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:59.543551849 +0000 UTC m=+3869.488411357" watchObservedRunningTime="2026-07-17 13:26:59.54491564 +0000 UTC m=+3869.489775143" Jul 17 13:26:59.989416 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:26:59.989380 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:01.005263 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:01.005226 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:01.530004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:01.529973 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:01.531000 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:01.530980 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:07.559437 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.559405 2568 generic.go:356] "Generic (PLEG): container finished" podID="18e30f56-95c3-467d-a134-9dad4883a12e" containerID="e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96" exitCode=0 Jul 17 13:27:07.560000 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.559455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerDied","Data":"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96"} Jul 17 13:27:07.560000 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.559982 2568 scope.go:117] "RemoveContainer" containerID="e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96" Jul 17 13:27:07.561038 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.561005 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:07.561141 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.561098 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:07.561207 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:07.561166 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:08.493108 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:08.493069 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:27:08.566702 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:08.566649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerStarted","Data":"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb"} Jul 17 13:27:08.921216 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:08.921185 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:27:08.921592 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:08.921567 2568 patch_prober.go:28] interesting pod/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 17 13:27:08.921787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:08.921760 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 17 13:27:09.919916 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:09.919875 2568 patch_prober.go:28] interesting pod/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 17 13:27:09.920290 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:09.919935 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 17 13:27:10.919519 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:10.919470 2568 patch_prober.go:28] interesting pod/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 17 13:27:10.919725 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:10.919542 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 17 13:27:11.919668 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:11.919632 2568 patch_prober.go:28] interesting pod/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 17 13:27:11.920187 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:11.919710 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 17 13:27:12.923243 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:12.923216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:27:12.923585 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:12.923481 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:27:12.924208 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:12.924187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:27:13.596653 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.596621 2568 generic.go:356] "Generic (PLEG): container finished" podID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerID="6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a" exitCode=0 Jul 17 13:27:13.596838 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.596706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerDied","Data":"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a"} Jul 17 13:27:13.597224 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.597205 2568 scope.go:117] "RemoveContainer" containerID="6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a" Jul 17 13:27:13.598153 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.598128 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:13.598203 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.598192 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:13.598241 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:13.598221 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 13:27:14.608959 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:14.608920 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerStarted","Data":"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1"} Jul 17 13:27:14.989903 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:14.989865 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:14.991658 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:14.991296 2568 patch_prober.go:28] interesting pod/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 17 13:27:14.991658 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:14.991375 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 17 13:27:15.989885 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:15.989846 2568 patch_prober.go:28] interesting pod/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 17 13:27:15.990621 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:15.990583 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 17 13:27:16.530800 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:16.530763 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:16.989884 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:16.989836 2568 patch_prober.go:28] interesting pod/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 17 13:27:16.990407 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:16.989922 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 17 13:27:17.990009 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:17.989972 2568 patch_prober.go:28] interesting pod/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 17 13:27:17.990626 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:17.990584 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 17 13:27:18.993641 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:18.993608 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:18.994755 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:18.994739 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:27:21.464070 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.464035 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:27:21.464448 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.464403 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="06094b63-f0a1-4401-8fcb-96f79ca6ee0c" containerName="istio-proxy" Jul 17 13:27:21.467499 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.467483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.469997 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.469971 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"kube-root-ca.crt\"" Jul 17 13:27:21.470943 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.470926 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"openshift-service-ca.crt\"" Jul 17 13:27:21.471043 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.470986 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"auth-test-model-e92eab-kserve-self-signed-certs\"" Jul 17 13:27:21.471043 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.470993 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"default-dockercfg-qwfzb\"" Jul 17 13:27:21.477184 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.477163 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:27:21.582061 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.582244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.582244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.582244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.582244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdk5\" (UniqueName: \"kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.582244 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.582192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683076 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdk5\" (UniqueName: \"kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683516 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683581 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683538 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.683581 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.683561 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.685453 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.685436 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.685746 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.685727 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.692039 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.692019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdk5\" (UniqueName: \"kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5\") pod \"auth-test-model-e92eab-kserve-754c7f955f-dl269\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.777538 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.777449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:21.907116 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:21.907090 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:27:21.908857 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:27:21.908828 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80dfd39_8c01_4367_9c41_bf25bd801d5b.slice/crio-85c393e63900a34948c5daa390fe89306c31ea25f53096a18dbde285c3083997 WatchSource:0}: Error finding container 85c393e63900a34948c5daa390fe89306c31ea25f53096a18dbde285c3083997: Status 404 returned error can't find the container with id 85c393e63900a34948c5daa390fe89306c31ea25f53096a18dbde285c3083997 Jul 17 13:27:22.645369 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:22.645334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" event={"ID":"b80dfd39-8c01-4367-9c41-bf25bd801d5b","Type":"ContainerStarted","Data":"d1ccd1df26808296133d9fb6da06ae37fe5525fffbec47e311ca88e3a3a6addf"} Jul 17 13:27:22.645873 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:22.645383 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" event={"ID":"b80dfd39-8c01-4367-9c41-bf25bd801d5b","Type":"ContainerStarted","Data":"85c393e63900a34948c5daa390fe89306c31ea25f53096a18dbde285c3083997"} Jul 17 13:27:22.663514 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:22.663470 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" podStartSLOduration=1.663455392 podStartE2EDuration="1.663455392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:22.661810671 +0000 UTC m=+3892.606670177" watchObservedRunningTime="2026-07-17 13:27:22.663455392 +0000 UTC m=+3892.608314894" Jul 17 13:27:31.778436 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:31.778388 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:31.778436 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:31.778450 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:31.790770 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:31.790744 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:27:32.700646 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:27:32.700615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:29:48.143274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:48.143234 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:29:53.762205 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.762109 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:29:53.765491 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.765473 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.767841 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.767823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"sub-test-model-e92eab-kserve-self-signed-certs\"" Jul 17 13:29:53.777224 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.777199 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:29:53.923389 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.923553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.923553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.923553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923473 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgn7\" (UniqueName: \"kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.923553 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:53.923717 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:53.923572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024072 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.023991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024072 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024072 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024341 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024341 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024341 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgn7\" (UniqueName: \"kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024512 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024391 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024512 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024453 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.024512 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.024480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.026391 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.026369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.026498 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.026480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.031932 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.031910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgn7\" (UniqueName: \"kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7\") pod \"sub-test-model-e92eab-kserve-6f5c746f75-g4j4p\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.074055 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.074028 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:29:54.200669 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.200635 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:29:54.203440 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:29:54.203409 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod375f1356_3199_4f2b_9c7c_ae04f0650cff.slice/crio-31a1c209dce60e22af793359a3718a3dc3ed5b4305f384608176128107da7b54 WatchSource:0}: Error finding container 31a1c209dce60e22af793359a3718a3dc3ed5b4305f384608176128107da7b54: Status 404 returned error can't find the container with id 31a1c209dce60e22af793359a3718a3dc3ed5b4305f384608176128107da7b54 Jul 17 13:29:54.338749 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.338646 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" event={"ID":"375f1356-3199-4f2b-9c7c-ae04f0650cff","Type":"ContainerStarted","Data":"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079"} Jul 17 13:29:54.338749 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.338711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" event={"ID":"375f1356-3199-4f2b-9c7c-ae04f0650cff","Type":"ContainerStarted","Data":"31a1c209dce60e22af793359a3718a3dc3ed5b4305f384608176128107da7b54"} Jul 17 13:29:54.357979 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:29:54.357938 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" podStartSLOduration=1.357919678 podStartE2EDuration="1.357919678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:54.356947375 +0000 UTC m=+4044.301806877" watchObservedRunningTime="2026-07-17 13:29:54.357919678 +0000 UTC m=+4044.302779181" Jul 17 13:30:03.163388 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.163325 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" podUID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" containerName="main" containerID="cri-o://d1ccd1df26808296133d9fb6da06ae37fe5525fffbec47e311ca88e3a3a6addf" gracePeriod=45 Jul 17 13:30:03.386022 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.385994 2568 generic.go:356] "Generic (PLEG): container finished" podID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" containerID="d1ccd1df26808296133d9fb6da06ae37fe5525fffbec47e311ca88e3a3a6addf" exitCode=0 Jul 17 13:30:03.386210 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.386066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" event={"ID":"b80dfd39-8c01-4367-9c41-bf25bd801d5b","Type":"ContainerDied","Data":"d1ccd1df26808296133d9fb6da06ae37fe5525fffbec47e311ca88e3a3a6addf"} Jul 17 13:30:03.418103 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.418046 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:30:03.515570 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515533 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.515787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515598 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzdk5\" (UniqueName: \"kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.515787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515619 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.515787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515653 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.515787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515672 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.515787 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515708 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs\") pod \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\" (UID: \"b80dfd39-8c01-4367-9c41-bf25bd801d5b\") " Jul 17 13:30:03.516069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515970 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:30:03.516069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.515987 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home" (OuterVolumeSpecName: "home") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:30:03.516069 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.516002 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:30:03.518007 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.517978 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5" (OuterVolumeSpecName: "kube-api-access-dzdk5") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "kube-api-access-dzdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:30:03.518098 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.518031 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:30:03.518098 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.518058 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm" (OuterVolumeSpecName: "dshm") pod "b80dfd39-8c01-4367-9c41-bf25bd801d5b" (UID: "b80dfd39-8c01-4367-9c41-bf25bd801d5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:30:03.616617 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616588 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:03.616617 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616612 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:03.616617 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616622 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdk5\" (UniqueName: \"kubernetes.io/projected/b80dfd39-8c01-4367-9c41-bf25bd801d5b-kube-api-access-dzdk5\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:03.616856 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616650 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:03.616856 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616659 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:03.616856 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:03.616667 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80dfd39-8c01-4367-9c41-bf25bd801d5b-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:30:04.074799 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.074763 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:30:04.074799 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.074808 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:30:04.086956 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.086928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:30:04.391718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.391627 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" Jul 17 13:30:04.391718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.391669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269" event={"ID":"b80dfd39-8c01-4367-9c41-bf25bd801d5b","Type":"ContainerDied","Data":"85c393e63900a34948c5daa390fe89306c31ea25f53096a18dbde285c3083997"} Jul 17 13:30:04.392182 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.391741 2568 scope.go:117] "RemoveContainer" containerID="d1ccd1df26808296133d9fb6da06ae37fe5525fffbec47e311ca88e3a3a6addf" Jul 17 13:30:04.404291 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.404271 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:30:04.413015 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.412989 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:30:04.416610 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.416590 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/auth-test-model-e92eab-kserve-754c7f955f-dl269"] Jul 17 13:30:04.586467 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:04.586437 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" path="/var/lib/kubelet/pods/b80dfd39-8c01-4367-9c41-bf25bd801d5b/volumes" Jul 17 13:30:15.375426 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.375389 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:30:15.375812 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.375757 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" containerName="main" Jul 17 13:30:15.375812 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.375768 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" containerName="main" Jul 17 13:30:15.375883 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.375824 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80dfd39-8c01-4367-9c41-bf25bd801d5b" containerName="main" Jul 17 13:30:15.380126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.380109 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.383730 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.383707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"default-dockercfg-dl6rb\"" Jul 17 13:30:15.383860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.383708 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"sub-test-model-5c3765-kserve-self-signed-certs\"" Jul 17 13:30:15.383860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.383751 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"openshift-service-ca.crt\"" Jul 17 13:30:15.383860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.383712 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"kube-root-ca.crt\"" Jul 17 13:30:15.387953 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.387933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:30:15.430733 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.430904 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.430904 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430796 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.430904 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.430904 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4lc\" (UniqueName: \"kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.431081 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.430941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531386 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531569 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531569 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531569 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531569 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531569 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531525 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4lc\" (UniqueName: \"kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531874 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531825 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.531952 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.531918 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.533892 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.533871 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.534068 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.534045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.542477 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.542455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4lc\" (UniqueName: \"kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc\") pod \"sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.689565 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.689527 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:15.820652 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:15.820628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:30:15.822050 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:30:15.822028 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod749e3666_a722_4ea6_8f2f_ae6cb7755b6f.slice/crio-c4ac73d1d940fc373b549a826f4499c83deb4d7f6253ed53a4f8144e0e0e9356 WatchSource:0}: Error finding container c4ac73d1d940fc373b549a826f4499c83deb4d7f6253ed53a4f8144e0e0e9356: Status 404 returned error can't find the container with id c4ac73d1d940fc373b549a826f4499c83deb4d7f6253ed53a4f8144e0e0e9356 Jul 17 13:30:16.448236 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:16.448197 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" event={"ID":"749e3666-a722-4ea6-8f2f-ae6cb7755b6f","Type":"ContainerStarted","Data":"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397"} Jul 17 13:30:16.448236 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:16.448248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" event={"ID":"749e3666-a722-4ea6-8f2f-ae6cb7755b6f","Type":"ContainerStarted","Data":"c4ac73d1d940fc373b549a826f4499c83deb4d7f6253ed53a4f8144e0e0e9356"} Jul 17 13:30:16.466357 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:16.466217 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" podStartSLOduration=1.466198066 podStartE2EDuration="1.466198066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:30:16.466066894 +0000 UTC m=+4066.410926402" watchObservedRunningTime="2026-07-17 13:30:16.466198066 +0000 UTC m=+4066.411057573" Jul 17 13:30:25.690297 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:25.690266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:25.690716 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:25.690315 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:25.702471 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:25.702441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:26.508227 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:26.508194 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:30:56.903608 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:56.903574 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:30:57.843054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:30:57.843023 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:31:11.923485 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:11.923442 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" podUID="375f1356-3199-4f2b-9c7c-ae04f0650cff" containerName="main" containerID="cri-o://df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079" gracePeriod=45 Jul 17 13:31:12.173106 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.173079 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:31:12.193438 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193356 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.193438 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193397 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.193438 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193424 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.193728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193453 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztgn7\" (UniqueName: \"kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.193728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193481 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.193728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.193570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir\") pod \"375f1356-3199-4f2b-9c7c-ae04f0650cff\" (UID: \"375f1356-3199-4f2b-9c7c-ae04f0650cff\") " Jul 17 13:31:12.194248 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.194051 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home" (OuterVolumeSpecName: "home") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:12.194395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.194170 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache" (OuterVolumeSpecName: "model-cache") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:12.194395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.194352 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:12.195843 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.195799 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm" (OuterVolumeSpecName: "dshm") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:12.195843 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.195831 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7" (OuterVolumeSpecName: "kube-api-access-ztgn7") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "kube-api-access-ztgn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:31:12.195984 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.195885 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "375f1356-3199-4f2b-9c7c-ae04f0650cff" (UID: "375f1356-3199-4f2b-9c7c-ae04f0650cff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:31:12.294339 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294305 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.294339 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294332 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.294339 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294340 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375f1356-3199-4f2b-9c7c-ae04f0650cff-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.294339 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294347 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.294600 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294357 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgn7\" (UniqueName: \"kubernetes.io/projected/375f1356-3199-4f2b-9c7c-ae04f0650cff-kube-api-access-ztgn7\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.294600 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.294364 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375f1356-3199-4f2b-9c7c-ae04f0650cff-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:12.710191 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.710159 2568 generic.go:356] "Generic (PLEG): container finished" podID="375f1356-3199-4f2b-9c7c-ae04f0650cff" containerID="df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079" exitCode=0 Jul 17 13:31:12.710395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.710210 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" Jul 17 13:31:12.710395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.710216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" event={"ID":"375f1356-3199-4f2b-9c7c-ae04f0650cff","Type":"ContainerDied","Data":"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079"} Jul 17 13:31:12.710395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.710327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p" event={"ID":"375f1356-3199-4f2b-9c7c-ae04f0650cff","Type":"ContainerDied","Data":"31a1c209dce60e22af793359a3718a3dc3ed5b4305f384608176128107da7b54"} Jul 17 13:31:12.710395 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.710347 2568 scope.go:117] "RemoveContainer" containerID="df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079" Jul 17 13:31:12.719037 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.719017 2568 scope.go:117] "RemoveContainer" containerID="df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079" Jul 17 13:31:12.719322 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:31:12.719301 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079\": container with ID starting with df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079 not found: ID does not exist" containerID="df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079" Jul 17 13:31:12.719394 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.719328 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079"} err="failed to get container status \"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079\": rpc error: code = NotFound desc = could not find container \"df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079\": container with ID starting with df96460f07f3eb632ac617c08587daed8588b0cf829eb7e756f00a5005503079 not found: ID does not exist" Jul 17 13:31:12.726967 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.726938 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:31:12.733079 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.733056 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/sub-test-model-e92eab-kserve-6f5c746f75-g4j4p"] Jul 17 13:31:12.863179 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:12.863067 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" podUID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" containerName="main" containerID="cri-o://761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397" gracePeriod=45 Jul 17 13:31:13.111267 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.111242 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:31:13.201335 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201303 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201529 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201347 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201529 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201397 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm4lc\" (UniqueName: \"kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201529 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201421 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201529 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201455 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201529 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201483 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm\") pod \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\" (UID: \"749e3666-a722-4ea6-8f2f-ae6cb7755b6f\") " Jul 17 13:31:13.201819 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201650 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:13.201819 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201743 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home" (OuterVolumeSpecName: "home") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:13.201915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201885 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:13.201966 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201908 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.201966 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.201930 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.203759 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.203740 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm" (OuterVolumeSpecName: "dshm") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:31:13.203836 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.203784 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc" (OuterVolumeSpecName: "kube-api-access-sm4lc") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "kube-api-access-sm4lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:31:13.203898 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.203866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "749e3666-a722-4ea6-8f2f-ae6cb7755b6f" (UID: "749e3666-a722-4ea6-8f2f-ae6cb7755b6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:31:13.302752 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.302718 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.302752 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.302749 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4lc\" (UniqueName: \"kubernetes.io/projected/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-kube-api-access-sm4lc\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.302752 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.302763 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.302987 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.302776 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/749e3666-a722-4ea6-8f2f-ae6cb7755b6f-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:31:13.715991 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.715957 2568 generic.go:356] "Generic (PLEG): container finished" podID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" containerID="761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397" exitCode=0 Jul 17 13:31:13.716143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.716003 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" Jul 17 13:31:13.716143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.716047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" event={"ID":"749e3666-a722-4ea6-8f2f-ae6cb7755b6f","Type":"ContainerDied","Data":"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397"} Jul 17 13:31:13.716143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.716097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs" event={"ID":"749e3666-a722-4ea6-8f2f-ae6cb7755b6f","Type":"ContainerDied","Data":"c4ac73d1d940fc373b549a826f4499c83deb4d7f6253ed53a4f8144e0e0e9356"} Jul 17 13:31:13.716143 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.716116 2568 scope.go:117] "RemoveContainer" containerID="761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397" Jul 17 13:31:13.724737 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.724712 2568 scope.go:117] "RemoveContainer" containerID="761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397" Jul 17 13:31:13.725011 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:31:13.724994 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397\": container with ID starting with 761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397 not found: ID does not exist" containerID="761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397" Jul 17 13:31:13.725063 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.725020 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397"} err="failed to get container status \"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397\": rpc error: code = NotFound desc = could not find container \"761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397\": container with ID starting with 761ab9a055b303235db40977504f0adebc1999b7cb141158b5ca2fcda8786397 not found: ID does not exist" Jul 17 13:31:13.734948 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.734923 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:31:13.742747 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:13.740778 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/sub-test-model-5c3765-kserve-74d46ccbfc-gh9xs"] Jul 17 13:31:14.586872 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:14.586828 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="375f1356-3199-4f2b-9c7c-ae04f0650cff" path="/var/lib/kubelet/pods/375f1356-3199-4f2b-9c7c-ae04f0650cff/volumes" Jul 17 13:31:14.587409 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:14.587389 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" path="/var/lib/kubelet/pods/749e3666-a722-4ea6-8f2f-ae6cb7755b6f/volumes" Jul 17 13:31:31.272853 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.272764 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273120 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375f1356-3199-4f2b-9c7c-ae04f0650cff" containerName="main" Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273131 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="375f1356-3199-4f2b-9c7c-ae04f0650cff" containerName="main" Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273149 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" containerName="main" Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273156 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" containerName="main" Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273204 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="749e3666-a722-4ea6-8f2f-ae6cb7755b6f" containerName="main" Jul 17 13:31:31.273242 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.273214 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="375f1356-3199-4f2b-9c7c-ae04f0650cff" containerName="main" Jul 17 13:31:31.277448 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.277427 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.280760 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.280734 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"openshift-service-ca.crt\"" Jul 17 13:31:31.280891 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.280763 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"rate-test-model-5c3765-kserve-self-signed-certs\"" Jul 17 13:31:31.280891 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.280769 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"default-dockercfg-dl6rb\"" Jul 17 13:31:31.281028 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.281015 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"kube-root-ca.crt\"" Jul 17 13:31:31.285676 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.285651 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:31:31.372152 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284bt\" (UniqueName: \"kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.372344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.372344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.372344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.372344 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372309 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.372531 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.372394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473481 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284bt\" (UniqueName: \"kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473481 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473784 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473784 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473784 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473784 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.473984 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.474040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.473991 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.474040 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.474019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.476083 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.476065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.476276 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.476261 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.482258 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.482232 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-284bt\" (UniqueName: \"kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt\") pod \"rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.588063 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.587961 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:31.726789 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.726763 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:31:31.728777 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:31:31.728736 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd05c1e2b_c139_4f87_a539_0d02772aa273.slice/crio-14a1741de24b60c34d809082ca23f4b3bc829ac7f311e3da4f8c12c2826303bd WatchSource:0}: Error finding container 14a1741de24b60c34d809082ca23f4b3bc829ac7f311e3da4f8c12c2826303bd: Status 404 returned error can't find the container with id 14a1741de24b60c34d809082ca23f4b3bc829ac7f311e3da4f8c12c2826303bd Jul 17 13:31:31.808440 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.808403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" event={"ID":"d05c1e2b-c139-4f87-a539-0d02772aa273","Type":"ContainerStarted","Data":"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7"} Jul 17 13:31:31.808594 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.808457 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" event={"ID":"d05c1e2b-c139-4f87-a539-0d02772aa273","Type":"ContainerStarted","Data":"14a1741de24b60c34d809082ca23f4b3bc829ac7f311e3da4f8c12c2826303bd"} Jul 17 13:31:31.829057 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:31.829002 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" podStartSLOduration=0.828984189 podStartE2EDuration="828.984189ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:31.82780579 +0000 UTC m=+4141.772665296" watchObservedRunningTime="2026-07-17 13:31:31.828984189 +0000 UTC m=+4141.773843692" Jul 17 13:31:41.588823 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:41.588782 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:41.589230 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:41.588837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:41.601274 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:41.601246 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:31:41.863444 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:31:41.863361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:32:14.543403 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:14.543368 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:32:29.563898 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.563856 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" podUID="d05c1e2b-c139-4f87-a539-0d02772aa273" containerName="main" containerID="cri-o://45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7" gracePeriod=45 Jul 17 13:32:29.821995 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.821920 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:32:29.855951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.855911 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.855951 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.855956 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.856225 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856015 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.856225 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856068 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-284bt\" (UniqueName: \"kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.856225 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856185 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.856400 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856244 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir\") pod \"d05c1e2b-c139-4f87-a539-0d02772aa273\" (UID: \"d05c1e2b-c139-4f87-a539-0d02772aa273\") " Jul 17 13:32:29.856400 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856232 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache" (OuterVolumeSpecName: "model-cache") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:32:29.856400 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856289 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home" (OuterVolumeSpecName: "home") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:32:29.856557 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856498 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:32:29.856663 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856642 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-tmp-dir\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:29.856760 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856667 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-model-cache\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:29.856760 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.856679 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-home\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:29.858435 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.858396 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm" (OuterVolumeSpecName: "dshm") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:32:29.858575 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.858551 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 13:32:29.858674 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.858657 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt" (OuterVolumeSpecName: "kube-api-access-284bt") pod "d05c1e2b-c139-4f87-a539-0d02772aa273" (UID: "d05c1e2b-c139-4f87-a539-0d02772aa273"). InnerVolumeSpecName "kube-api-access-284bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:32:29.957582 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.957538 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d05c1e2b-c139-4f87-a539-0d02772aa273-dshm\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:29.957582 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.957574 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d05c1e2b-c139-4f87-a539-0d02772aa273-tls-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:29.957582 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:29.957588 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284bt\" (UniqueName: \"kubernetes.io/projected/d05c1e2b-c139-4f87-a539-0d02772aa273-kube-api-access-284bt\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:32:30.072728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.072601 2568 generic.go:356] "Generic (PLEG): container finished" podID="d05c1e2b-c139-4f87-a539-0d02772aa273" containerID="45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7" exitCode=0 Jul 17 13:32:30.072728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.072647 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" Jul 17 13:32:30.072728 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.072666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" event={"ID":"d05c1e2b-c139-4f87-a539-0d02772aa273","Type":"ContainerDied","Data":"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7"} Jul 17 13:32:30.073022 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.072735 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk" event={"ID":"d05c1e2b-c139-4f87-a539-0d02772aa273","Type":"ContainerDied","Data":"14a1741de24b60c34d809082ca23f4b3bc829ac7f311e3da4f8c12c2826303bd"} Jul 17 13:32:30.073022 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.072758 2568 scope.go:117] "RemoveContainer" containerID="45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7" Jul 17 13:32:30.082885 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.082862 2568 scope.go:117] "RemoveContainer" containerID="45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7" Jul 17 13:32:30.083232 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:32:30.083212 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7\": container with ID starting with 45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7 not found: ID does not exist" containerID="45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7" Jul 17 13:32:30.083318 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.083240 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7"} err="failed to get container status \"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7\": rpc error: code = NotFound desc = could not find container \"45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7\": container with ID starting with 45d8a4a21f5e7be79bec5e0ce04e1a6dcdde905c1e9ec48f1075f96d6eaff5e7 not found: ID does not exist" Jul 17 13:32:30.094659 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.094622 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:32:30.099034 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.099004 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/rate-test-model-5c3765-kserve-7f6865bfc5-6lnhk"] Jul 17 13:32:30.587211 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:30.587174 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d05c1e2b-c139-4f87-a539-0d02772aa273" path="/var/lib/kubelet/pods/d05c1e2b-c139-4f87-a539-0d02772aa273/volumes" Jul 17 13:32:59.696384 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:59.696295 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:32:59.696870 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:32:59.696520 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" containerID="cri-o://b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb" gracePeriod=30 Jul 17 13:33:04.949860 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:04.949827 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:33:05.001957 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.001859 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.001957 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.001924 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.001957 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.001955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.001971 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002022 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002043 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002075 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002239 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002106 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs\") pod \"18e30f56-95c3-467d-a134-9dad4883a12e\" (UID: \"18e30f56-95c3-467d-a134-9dad4883a12e\") " Jul 17 13:33:05.002541 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002277 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:05.002541 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002451 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.002541 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002491 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:05.002722 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002555 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:05.002790 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.002776 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:33:05.003385 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.003353 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data" (OuterVolumeSpecName: "istio-data") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:05.004380 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.004349 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:05.004494 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.004373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:33:05.004851 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.004830 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw" (OuterVolumeSpecName: "kube-api-access-sphtw") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "kube-api-access-sphtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:33:05.004934 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.004844 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token" (OuterVolumeSpecName: "istio-token") pod "18e30f56-95c3-467d-a134-9dad4883a12e" (UID: "18e30f56-95c3-467d-a134-9dad4883a12e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:33:05.103548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103493 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e30f56-95c3-467d-a134-9dad4883a12e-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103539 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103548 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103556 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103548 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103563 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e30f56-95c3-467d-a134-9dad4883a12e-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103573 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103581 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e30f56-95c3-467d-a134-9dad4883a12e-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.103915 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.103590 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphtw\" (UniqueName: \"kubernetes.io/projected/18e30f56-95c3-467d-a134-9dad4883a12e-kube-api-access-sphtw\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:05.237147 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.237110 2568 generic.go:356] "Generic (PLEG): container finished" podID="18e30f56-95c3-467d-a134-9dad4883a12e" containerID="b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb" exitCode=0 Jul 17 13:33:05.237147 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.237149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerDied","Data":"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb"} Jul 17 13:33:05.237372 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.237161 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" Jul 17 13:33:05.237372 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.237194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg" event={"ID":"18e30f56-95c3-467d-a134-9dad4883a12e","Type":"ContainerDied","Data":"5dd75318fa0e2a58a2498fcde4be416adb3671b7e3b1611853d5369c7a501d3e"} Jul 17 13:33:05.237372 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.237216 2568 scope.go:117] "RemoveContainer" containerID="b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb" Jul 17 13:33:05.246862 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.246839 2568 scope.go:117] "RemoveContainer" containerID="e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96" Jul 17 13:33:05.255216 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.255193 2568 scope.go:117] "RemoveContainer" containerID="b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb" Jul 17 13:33:05.255540 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:33:05.255523 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb\": container with ID starting with b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb not found: ID does not exist" containerID="b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb" Jul 17 13:33:05.255594 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.255549 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb"} err="failed to get container status \"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb\": rpc error: code = NotFound desc = could not find container \"b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb\": container with ID starting with b280ac23bfd5bb539b27b4c506353f7412f509d2aed28aba2430acc2f2b833fb not found: ID does not exist" Jul 17 13:33:05.255594 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.255568 2568 scope.go:117] "RemoveContainer" containerID="e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96" Jul 17 13:33:05.255827 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:33:05.255808 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96\": container with ID starting with e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96 not found: ID does not exist" containerID="e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96" Jul 17 13:33:05.255877 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.255831 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96"} err="failed to get container status \"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96\": rpc error: code = NotFound desc = could not find container \"e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96\": container with ID starting with e5830b15c24cb879250305a67245458b1064c34acfa80df9b272e699531dec96 not found: ID does not exist" Jul 17 13:33:05.258357 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.258332 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:33:05.262326 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:05.262302 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e92eab-openshift-default-76f9cb4f8c-p6qjg"] Jul 17 13:33:06.587120 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:06.587083 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" path="/var/lib/kubelet/pods/18e30f56-95c3-467d-a134-9dad4883a12e/volumes" Jul 17 13:33:25.820105 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:25.820062 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:33:25.820671 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:25.820351 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" containerID="cri-o://5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1" gracePeriod=30 Jul 17 13:33:26.502583 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:26.502543 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-778dd9b68-kvf29_98143c94-1355-49e8-9e17-34a18e746a62/authorino/0.log" Jul 17 13:33:31.062348 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.062314 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:33:31.150848 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.150812 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.150848 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.150850 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151053 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.150891 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151104 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151079 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151175 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151157 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151234 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151213 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151297 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151242 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151297 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151400 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151326 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpc7z\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z\") pod \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\" (UID: \"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3\") " Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151527 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151499 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151640 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151740 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151761 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-credential-socket\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.151796 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.151781 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:31.152065 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.152026 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:31.153285 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.153260 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 13:33:31.153669 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.153644 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z" (OuterVolumeSpecName: "kube-api-access-mpc7z") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "kube-api-access-mpc7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:33:31.153758 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.153677 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 13:33:31.153758 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.153728 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" (UID: "1ab91a10-eaac-44a5-9a31-a1e0c47de0f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 13:33:31.253071 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253028 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istiod-ca-cert\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253071 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253060 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-workload-certs\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253071 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253068 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpc7z\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-kube-api-access-mpc7z\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253071 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253078 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-podinfo\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253071 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253086 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-token\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253380 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253094 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-envoy\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.253380 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.253103 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3-istio-data\") on node \"ip-10-0-129-112.ec2.internal\" DevicePath \"\"" Jul 17 13:33:31.353075 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.353043 2568 generic.go:356] "Generic (PLEG): container finished" podID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerID="5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1" exitCode=0 Jul 17 13:33:31.353283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.353091 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" Jul 17 13:33:31.353283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.353128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerDied","Data":"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1"} Jul 17 13:33:31.353283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.353194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj" event={"ID":"1ab91a10-eaac-44a5-9a31-a1e0c47de0f3","Type":"ContainerDied","Data":"aa04c77f9a6ff3bc9a5ee6d54f0251fe91836fa5684641af12647c04c581d89b"} Jul 17 13:33:31.353283 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.353224 2568 scope.go:117] "RemoveContainer" containerID="5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1" Jul 17 13:33:31.362574 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.362553 2568 scope.go:117] "RemoveContainer" containerID="6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a" Jul 17 13:33:31.371340 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.371316 2568 scope.go:117] "RemoveContainer" containerID="5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1" Jul 17 13:33:31.371643 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:33:31.371624 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1\": container with ID starting with 5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1 not found: ID does not exist" containerID="5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1" Jul 17 13:33:31.371707 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.371656 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1"} err="failed to get container status \"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1\": rpc error: code = NotFound desc = could not find container \"5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1\": container with ID starting with 5f98e9686871f0bd0260fd875eec2e4a0894cce08149e6aa32f75dcdd90081c1 not found: ID does not exist" Jul 17 13:33:31.371707 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.371675 2568 scope.go:117] "RemoveContainer" containerID="6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a" Jul 17 13:33:31.372049 ip-10-0-129-112 kubenswrapper[2568]: E0717 13:33:31.372029 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a\": container with ID starting with 6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a not found: ID does not exist" containerID="6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a" Jul 17 13:33:31.372114 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.372058 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a"} err="failed to get container status \"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a\": rpc error: code = NotFound desc = could not find container \"6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a\": container with ID starting with 6baa246e80af3238e75e8ef81994efe59dff3378a8255842f0725ef924fcb74a not found: ID does not exist" Jul 17 13:33:31.375786 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.375744 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:33:31.382173 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:31.382140 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5c3765-openshift-default-569658788c-h4cgj"] Jul 17 13:33:32.372122 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:32.372077 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lsnqt_04716e68-f3ee-42ca-9778-c2f34c791ac4/manager/0.log" Jul 17 13:33:32.587858 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:32.587816 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" path="/var/lib/kubelet/pods/1ab91a10-eaac-44a5-9a31-a1e0c47de0f3/volumes" Jul 17 13:33:32.760515 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:32.760480 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-sn6f7_62e054db-eec7-49e2-8646-b6666ac3bafa/copy-manifests/0.log" Jul 17 13:33:32.768084 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:32.768058 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-sn6f7_62e054db-eec7-49e2-8646-b6666ac3bafa/manager/0.log" Jul 17 13:33:36.168200 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:36.168166 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-778dd9b68-kvf29_98143c94-1355-49e8-9e17-34a18e746a62/authorino/0.log" Jul 17 13:33:37.394911 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:37.394882 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxw4h_b0b59f2a-c9d4-4335-9e56-838ecd19a270/istio-proxy/0.log" Jul 17 13:33:37.634555 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:37.634526 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-mg7pg_a8f6db39-82a4-4bff-9875-ab666896b5c2/kube-auth-proxy/0.log" Jul 17 13:33:37.873950 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:37.873848 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/1.log" Jul 17 13:33:38.229819 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:38.229786 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb9845cfd-95zm4_4a85a944-52d1-4a4b-a48f-08967187bf33/router/0.log" Jul 17 13:33:38.805715 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:38.805672 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-l7wxw_f8970582-873c-4c5f-8507-94bd61961614/main/0.log" Jul 17 13:33:39.028905 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:39.028873 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-8fddp_8eb299a6-2e05-435b-9b1b-503c4cc9381e/main/0.log" Jul 17 13:33:44.349301 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349261 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mj6w/must-gather-w2mnl"] Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349609 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349620 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349632 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349637 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349643 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d05c1e2b-c139-4f87-a539-0d02772aa273" containerName="main" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349649 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d05c1e2b-c139-4f87-a539-0d02772aa273" containerName="main" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349657 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349662 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349673 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349678 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.349773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349772 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d05c1e2b-c139-4f87-a539-0d02772aa273" containerName="main" Jul 17 13:33:44.350159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349783 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.350159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349789 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.350159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349794 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e30f56-95c3-467d-a134-9dad4883a12e" containerName="istio-proxy" Jul 17 13:33:44.350159 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.349902 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab91a10-eaac-44a5-9a31-a1e0c47de0f3" containerName="istio-proxy" Jul 17 13:33:44.354142 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.354121 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.360197 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.360173 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"kube-root-ca.crt\"" Jul 17 13:33:44.360870 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.360846 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"openshift-service-ca.crt\"" Jul 17 13:33:44.370065 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.370032 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/must-gather-w2mnl"] Jul 17 13:33:44.378490 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.378439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm87g\" (UniqueName: \"kubernetes.io/projected/e64ddd16-436c-463c-ac46-071d9d9ad944-kube-api-access-nm87g\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.378718 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.378542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64ddd16-436c-463c-ac46-071d9d9ad944-must-gather-output\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.479786 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.479741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64ddd16-436c-463c-ac46-071d9d9ad944-must-gather-output\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.479973 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.479828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm87g\" (UniqueName: \"kubernetes.io/projected/e64ddd16-436c-463c-ac46-071d9d9ad944-kube-api-access-nm87g\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.480088 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.480068 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64ddd16-436c-463c-ac46-071d9d9ad944-must-gather-output\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.491328 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.491302 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm87g\" (UniqueName: \"kubernetes.io/projected/e64ddd16-436c-463c-ac46-071d9d9ad944-kube-api-access-nm87g\") pod \"must-gather-w2mnl\" (UID: \"e64ddd16-436c-463c-ac46-071d9d9ad944\") " pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.663566 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.663530 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" Jul 17 13:33:44.801516 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.801487 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/must-gather-w2mnl"] Jul 17 13:33:44.803765 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:33:44.803738 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64ddd16_436c_463c_ac46_071d9d9ad944.slice/crio-4b357f9fbc8ff22e6b5f590f5389400ecb14229181c354116a9214904d0f5aed WatchSource:0}: Error finding container 4b357f9fbc8ff22e6b5f590f5389400ecb14229181c354116a9214904d0f5aed: Status 404 returned error can't find the container with id 4b357f9fbc8ff22e6b5f590f5389400ecb14229181c354116a9214904d0f5aed Jul 17 13:33:44.805629 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:44.805606 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 13:33:45.425621 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:45.425582 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" event={"ID":"e64ddd16-436c-463c-ac46-071d9d9ad944","Type":"ContainerStarted","Data":"4b357f9fbc8ff22e6b5f590f5389400ecb14229181c354116a9214904d0f5aed"} Jul 17 13:33:46.434650 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:46.434600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" event={"ID":"e64ddd16-436c-463c-ac46-071d9d9ad944","Type":"ContainerStarted","Data":"ab1cca33174eb311b498b31848656fd2facc4f1e55bda021bfd2093b7fc41cfc"} Jul 17 13:33:46.435198 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:46.434664 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" event={"ID":"e64ddd16-436c-463c-ac46-071d9d9ad944","Type":"ContainerStarted","Data":"10a414dce6395a544154df1052b2c89bf731bd6ce979fe39cc5dffa1c373edbb"} Jul 17 13:33:46.457758 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:46.457702 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mj6w/must-gather-w2mnl" podStartSLOduration=1.285885768 podStartE2EDuration="2.457664067s" totalImagesPullingTime="1.171778299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:33:46.455855778 +0000 UTC m=+4276.400715282" watchObservedRunningTime="2026-07-17 13:33:46.457664067 +0000 UTC m=+4276.402523570" Jul 17 13:33:47.770289 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:47.770255 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2n479_f7f9c9d3-e58c-4eb0-8319-ded8fdf4a4db/global-pull-secret-syncer/0.log" Jul 17 13:33:47.985352 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:47.985313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cp7zn_4daf38fd-9b5a-49b7-b1a7-87f8c365bb4a/konnectivity-agent/0.log" Jul 17 13:33:48.009126 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:48.009094 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-112.ec2.internal_b9e412b5125944006d4709c364d63da6/haproxy/0.log" Jul 17 13:33:53.567259 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:53.567205 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-778dd9b68-kvf29_98143c94-1355-49e8-9e17-34a18e746a62/authorino/0.log" Jul 17 13:33:55.665165 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:55.665126 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7v5bn_2c95494b-39b9-4e8e-bcf1-8d722cd56a6f/cluster-monitoring-operator/0.log" Jul 17 13:33:55.817430 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:55.817388 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-nhchd_eed48311-7549-43f3-ab61-4cd4a6d3e8a0/monitoring-plugin/0.log" Jul 17 13:33:56.037310 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.037208 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shxjx_520ca06e-980e-4431-9736-355c74e80b74/node-exporter/0.log" Jul 17 13:33:56.059054 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.059024 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shxjx_520ca06e-980e-4431-9736-355c74e80b74/kube-rbac-proxy/0.log" Jul 17 13:33:56.084004 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.083978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shxjx_520ca06e-980e-4431-9736-355c74e80b74/init-textfile/0.log" Jul 17 13:33:56.204026 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.203976 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/prometheus/0.log" Jul 17 13:33:56.221545 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.221518 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/config-reloader/0.log" Jul 17 13:33:56.245783 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.245751 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/thanos-sidecar/0.log" Jul 17 13:33:56.268627 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.268598 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/kube-rbac-proxy-web/0.log" Jul 17 13:33:56.291438 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.291358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/kube-rbac-proxy/0.log" Jul 17 13:33:56.323136 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.323088 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/kube-rbac-proxy-thanos/0.log" Jul 17 13:33:56.346887 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:56.346850 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bca664a3-30b5-4cce-9083-6cf5b9583620/init-config-reloader/0.log" Jul 17 13:33:58.251635 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.251589 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr"] Jul 17 13:33:58.257565 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.257530 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.261486 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.261455 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mj6w\"/\"default-dockercfg-lsxbf\"" Jul 17 13:33:58.265773 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.265746 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr"] Jul 17 13:33:58.336888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.336527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-sys\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.336888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.336588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-proc\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.336888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.336642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-podres\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.336888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.336676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-lib-modules\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.336888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.336748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrl2\" (UniqueName: \"kubernetes.io/projected/96099846-c5f5-4636-b1a9-eb471a51c69c-kube-api-access-tkrl2\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.437767 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.437727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-sys\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.437955 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.437940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-proc\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438035 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.437869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-sys\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438035 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.437992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-podres\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438147 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.438048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-proc\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438147 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.438091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-lib-modules\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438256 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.438216 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-podres\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438256 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.438223 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96099846-c5f5-4636-b1a9-eb471a51c69c-lib-modules\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.438256 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.438221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrl2\" (UniqueName: \"kubernetes.io/projected/96099846-c5f5-4636-b1a9-eb471a51c69c-kube-api-access-tkrl2\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.448449 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.448417 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrl2\" (UniqueName: \"kubernetes.io/projected/96099846-c5f5-4636-b1a9-eb471a51c69c-kube-api-access-tkrl2\") pod \"perf-node-gather-daemonset-jzzdr\" (UID: \"96099846-c5f5-4636-b1a9-eb471a51c69c\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.572636 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.572551 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:58.783530 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:58.783488 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr"] Jul 17 13:33:58.786890 ip-10-0-129-112 kubenswrapper[2568]: W0717 13:33:58.786849 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96099846_c5f5_4636_b1a9_eb471a51c69c.slice/crio-adf799f45179757c49f395ced61a5e7ef482d274f60004bac056eee271976099 WatchSource:0}: Error finding container adf799f45179757c49f395ced61a5e7ef482d274f60004bac056eee271976099: Status 404 returned error can't find the container with id adf799f45179757c49f395ced61a5e7ef482d274f60004bac056eee271976099 Jul 17 13:33:59.535261 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:59.535225 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" event={"ID":"96099846-c5f5-4636-b1a9-eb471a51c69c","Type":"ContainerStarted","Data":"836e1f13f693a43346f260d5176aec62ebc2b10864c8a7c4f1809ee03bdd1325"} Jul 17 13:33:59.535733 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:59.535288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" event={"ID":"96099846-c5f5-4636-b1a9-eb471a51c69c","Type":"ContainerStarted","Data":"adf799f45179757c49f395ced61a5e7ef482d274f60004bac056eee271976099"} Jul 17 13:33:59.535733 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:59.535310 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:33:59.550199 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:59.550146 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" podStartSLOduration=1.550128325 podStartE2EDuration="1.550128325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:33:59.549302208 +0000 UTC m=+4289.494161716" watchObservedRunningTime="2026-07-17 13:33:59.550128325 +0000 UTC m=+4289.494987874" Jul 17 13:33:59.712922 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:33:59.712885 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-9m95k_0fa0e48a-d489-44d1-a3e8-30d99a0b0066/volume-data-source-validator/0.log" Jul 17 13:34:00.830252 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:00.830224 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n85zn_4f8dba38-59ce-4c23-8e17-5a713a852153/dns/0.log" Jul 17 13:34:00.851417 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:00.851389 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n85zn_4f8dba38-59ce-4c23-8e17-5a713a852153/kube-rbac-proxy/0.log" Jul 17 13:34:00.873888 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:00.873859 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2nfht_5290d129-15a0-4b75-b59a-3878bf3e9e7b/dns-node-resolver/0.log" Jul 17 13:34:01.459034 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:01.459006 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6756c7dc49-pw49x_12a52a13-07ec-4c12-bf1d-698dc50ae6a2/registry/0.log" Jul 17 13:34:01.504488 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:01.504454 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvl9t_61b4f3f4-920d-417f-9cf0-d3d60b64a434/node-ca/0.log" Jul 17 13:34:02.473936 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:02.473899 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxw4h_b0b59f2a-c9d4-4335-9e56-838ecd19a270/istio-proxy/0.log" Jul 17 13:34:02.620795 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:02.620763 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-mg7pg_a8f6db39-82a4-4bff-9875-ab666896b5c2/kube-auth-proxy/0.log" Jul 17 13:34:02.671469 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:02.671434 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/0.log" Jul 17 13:34:02.709889 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:02.709852 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/1.log" Jul 17 13:34:02.784770 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:02.784654 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb9845cfd-95zm4_4a85a944-52d1-4a4b-a48f-08967187bf33/router/0.log" Jul 17 13:34:03.515042 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:03.515011 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rjdhb_28ea3db6-f0ed-449c-9ad6-4e1253290b6d/serve-healthcheck-canary/0.log" Jul 17 13:34:03.972135 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:03.972105 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cjcz9_6468c778-c914-484e-83a9-eae7fe6f52d0/insights-operator/0.log" Jul 17 13:34:03.973763 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:03.973732 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cjcz9_6468c778-c914-484e-83a9-eae7fe6f52d0/insights-operator/1.log" Jul 17 13:34:04.136029 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:04.136001 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27zj_ab7793de-4484-4e91-8ea9-6b32478a70fd/kube-rbac-proxy/0.log" Jul 17 13:34:04.157798 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:04.157767 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27zj_ab7793de-4484-4e91-8ea9-6b32478a70fd/exporter/0.log" Jul 17 13:34:04.179666 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:04.179639 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27zj_ab7793de-4484-4e91-8ea9-6b32478a70fd/extractor/0.log" Jul 17 13:34:05.553499 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:05.553470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-jzzdr" Jul 17 13:34:06.665107 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:06.665066 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lsnqt_04716e68-f3ee-42ca-9778-c2f34c791ac4/manager/0.log" Jul 17 13:34:06.850927 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:06.850897 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-sn6f7_62e054db-eec7-49e2-8646-b6666ac3bafa/manager/0.log" Jul 17 13:34:06.873677 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:06.873652 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-sn6f7_62e054db-eec7-49e2-8646-b6666ac3bafa/copy-manifests/0.log" Jul 17 13:34:08.269855 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:08.269827 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-774b4c8dcd-b9j69_05946b95-63c3-4717-b8d2-19a92a65cf15/manager/0.log" Jul 17 13:34:08.840752 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:08.840665 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxw4h_b0b59f2a-c9d4-4335-9e56-838ecd19a270/istio-proxy/0.log" Jul 17 13:34:08.960151 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:08.960113 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-mg7pg_a8f6db39-82a4-4bff-9875-ab666896b5c2/kube-auth-proxy/0.log" Jul 17 13:34:08.995827 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:08.995794 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/0.log" Jul 17 13:34:09.034907 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:09.034872 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/1.log" Jul 17 13:34:09.091453 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:09.091373 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb9845cfd-95zm4_4a85a944-52d1-4a4b-a48f-08967187bf33/router/0.log" Jul 17 13:34:10.349253 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:10.349176 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cxw4h_b0b59f2a-c9d4-4335-9e56-838ecd19a270/istio-proxy/0.log" Jul 17 13:34:10.472286 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:10.472256 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-mg7pg_a8f6db39-82a4-4bff-9875-ab666896b5c2/kube-auth-proxy/0.log" Jul 17 13:34:10.510449 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:10.510418 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/0.log" Jul 17 13:34:10.555589 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:10.555310 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-gqxlk_b4809646-3859-4172-ae02-d2ce581c3c10/istio-proxy/1.log" Jul 17 13:34:10.612277 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:10.612196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bb9845cfd-95zm4_4a85a944-52d1-4a4b-a48f-08967187bf33/router/0.log" Jul 17 13:34:13.848323 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:13.848287 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-kl8f8_37431f8a-bcda-4b4d-b8a2-6dc14a402a5f/kube-storage-version-migrator-operator/1.log" Jul 17 13:34:13.849304 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:13.849282 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-kl8f8_37431f8a-bcda-4b4d-b8a2-6dc14a402a5f/kube-storage-version-migrator-operator/0.log" Jul 17 13:34:14.819025 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.818997 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/kube-multus-additional-cni-plugins/0.log" Jul 17 13:34:14.841431 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.841407 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/egress-router-binary-copy/0.log" Jul 17 13:34:14.861793 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.861762 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/cni-plugins/0.log" Jul 17 13:34:14.882830 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.882806 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/bond-cni-plugin/0.log" Jul 17 13:34:14.902243 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.902217 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/routeoverride-cni/0.log" Jul 17 13:34:14.922494 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.922470 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/whereabouts-cni-bincopy/0.log" Jul 17 13:34:14.942801 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:14.942777 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6v7ml_ed35a9c8-6dc7-4572-b4ed-e5bb60a76ac0/whereabouts-cni/0.log" Jul 17 13:34:15.368762 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:15.368659 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sz2nb_f3f71dfb-6002-40d1-8004-9e57f9bcf384/kube-multus/0.log" Jul 17 13:34:15.442202 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:15.442175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6t2j_3ae54f7a-49e7-4701-a8e8-af4f183d4d09/network-metrics-daemon/0.log" Jul 17 13:34:15.463929 ip-10-0-129-112 kubenswrapper[2568]: I0717 13:34:15.463903 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6t2j_3ae54f7a-49e7-4701-a8e8-af4f183d4d09/kube-rbac-proxy/0.log"